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The  Research  and  Technology 
Organisation  (RTO)  of  NATO 


RTO  is  the  single  focus  in  NATO  for  Defence  Research  and  Technology  activities.  Its  mission  is  to  conduct  and  promote 
co-operative  research  and  information  exchange.  The  objective  is  to  support  the  development  and  effective  use  of 
national  defence  research  and  technology  and  to  meet  the  military  needs  of  the  Alliance,  to  maintain  a  technological 
lead,  and  to  provide  advice  to  NATO  and  national  decision  makers.  The  RTO  performs  its  mission  with  the  support  of  an 
extensive  network  of  national  experts.  It  also  ensures  effective  co-ordination  with  other  NATO  bodies  involved  in  R&T 
activities. 


RTO  reports  both  to  the  Military  Committee  of  NATO  and  to  the  Conference  of  National  Armament  Directors. 
It  comprises  a  Research  and  Technology  Board  (RTB)  as  the  highest  level  of  national  representation  and  the  Research 
and  Technology  Agency  (RTA),  a  dedicated  staff  with  its  headquarters  in  Neuilly,  near  Paris,  France.  In  order  to 
facilitate  contacts  with  the  military  users  and  other  NATO  activities,  a  small  part  of  the  RTA  staff  is  located  in  NATO 
Headquarters  in  Brussels.  The  Brussels  staff  also  co-ordinates  RTO’s  co-operation  with  nations  in  Middle  and  Eastern 
Europe,  to  which  RTO  attaches  particular  importance  especially  as  working  together  in  the  field  of  research  is  one  of  the 
more  promising  areas  of  co-operation. 


The  total  spectrum  of  R&T  activities  is  covered  by  the  following  7  bodies: 


•  AVT 

•  HFM 

•  1ST 

•  NMSG 

•  SAS 

•  SCI 

•  SET 


Applied  Vehicle  Technology  Panel 
Human  Factors  and  Medicine  Panel 
Information  Systems  Technology  Panel 
NATO  Modelling  and  Simulation  Group 
System  Analysis  and  Studies  Panel 
Systems  Concepts  and  Integration  Panel 
Sensors  and  Electronics  Technology  Panel 


These  bodies  are  made  up  of  national  representatives  as  well  as  generally  recognised  ‘world  class’  scientists.  They  also 
provide  a  communication  link  to  military  users  and  other  NATO  bodies.  RTO’s  scientific  and  technological  work  is 
carried  out  by  Technical  Teams,  created  for  specific  activities  and  with  a  specific  duration.  Such  Technical  Teams  can 
organise  workshops,  symposia,  field  trials,  lecture  series  and  training  courses.  An  important  function  of  these  Technical 
Teams  is  to  ensure  the  continuity  of  the  expert  networks. 

RTO  builds  upon  earlier  co-operation  in  defence  research  and  technology  as  set-up  under  the  Advisory  Group  for 
Aerospace  Research  and  Development  (AGARD)  and  the  Defence  Research  Group  (DRG).  AGARD  and  the  DRG  share 
common  roots  in  that  they  were  both  established  at  the  initiative  of  Dr  Theodore  von  Karman,  a  leading  aerospace 
scientist,  who  early  on  recognised  the  importance  of  scientific  support  for  the  Allied  Armed  Forces.  RTO  is  capitalising 
on  these  common  roots  in  order  to  provide  the  Alliance  and  the  NATO  nations  with  a  strong  scientific  and  technological 
basis  that  will  guarantee  a  solid  base  for  the  future. 
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Human  Behaviour  Representation 
in  Constructive  Modelling 
(RTO-MP-HFM- 143) 

Executive  Summary 

NATO  RTO  has  coordinated  a  number  of  studies  into  the  modelling  of  personnel  and  platforms  for  use  in 
military  simulations  (Dompke  &  Scheckeler,  1999;  RTO,  2001)  and  interest  in  human  modelling  continues 
in  a  number  of  NATO  RTO  panels.  RTO  HFM  RTG-128  on  Human  Behaviour  Representation  (HBR)  in 
Constructive  Simulation  is  in  the  process  of  providing  guidance  to  operational  analysts  and  engineers  on 
human  factors  that  can  be  included  in  operational  models.  Such  models  are  used  for  various  assessments  - 
notably  materiel,  mission  support  and  tactics  -  as  well  as  training.  The  goal  of  HFM  RTG-128  is  to  make 
recommendations  to  the  HBR  modelling  community  regarding  the  use  of  human  factors  in  constructive 
models  in  a  systematic  way,  paving  the  way  for  standardisation  and  re-use  of  modules  describing  specific 
human  factors. 

HFM-143/RSM  “Human  Behaviour  Representation  in  Constructive  Modelling”  was  initiated  by  RTG-128 
as  part  of  its  activities  to  further  the  field  of  human  modelling  for  NATO  M&S.  Thirty-three  people  with 
an  interest  in  modelling  human  behaviour  and  performance  from  seven  NATO  member  and  Partners- 
for-Peace  countries  gathered  at  the  Canadian  Forces  College  in  Toronto,  Canada  for  a  2-day  workshop 
(30-31  May  2007)  to  discuss,  debate  and  exchange  ideas  on  aspects  of  human  modelling. 

Each  day  started  with  a  keynote  presentation  on  HBR’s  role  in  simulation.  The  first  keynote  presentation, 
by  Dr.  Robert  Foster,  DDR&E,  provided  a  government  and  NATO  perspective,  outlining  the  need  to  go 
beyond  the  requirements  of  traditional  combat  modelling  to  include  the  broader  range  of  military  activities 
including  humanitarian  aid  and  nation  reconstruction,  focusing  to  a  greater  extent  on  the  human  aspects  of 
Effects  Based  Operations  and  non-kinetic  warfare.  The  second  keynote  presentation,  by  Mr.  Mike  Greenley, 
CAE  Inc.  provided  an  industry  perspective,  noting  the  need  for  best-practice  approaches  to  human  modelling 
that  evolve  naturally  rather  than  mandated  constraints  and  that  if  industry  perceives  a  viable  market,  such  as 
the  broader  training  of  personnel,  companies  will  participate  in  that  evolution  as  partners.  The  following 
seven  topic  areas  were  then  discussed  in  subsequent  sessions: 

1)  What  Human  Factors  Does  an  Operation  Involve? 

2)  Human  Task  Representation  in  M&S. 

3)  Behaviour  Generation:  Variability  and  Choice. 

4)  The  Concept  of  Moderators. 

5)  Militarily  Relevant  Mental  Output  Measures:  Workload,  Situation  Awareness  and  Other  Useful 
Concepts. 

6)  Complexity,  Hierarchy,  Modularity  and  Validity  in  HBR  Architectures. 

7)  From  Individual  to  Group  Behaviour. 

Each  session  comprised  some  introductory  remarks  by  the  session  chair  and  brief  presentations  by 
researchers  in  the  field  to  offer  perspectives  on  the  discussion  topic.  This  was  followed  by  a  general 
discussion  among  the  participants.  The  focus  on  plenary  discussion  with  short  presentations  seemed  to  be 
well  received  and  there  was  extensive  participation  in  the  discussions  by  all  of  the  attendees. 
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There  was  general  agreement  with  the  points  of  view  of  the  HFM-128  approach  to  documenting  the 
human  factors  of  HBR  and  that  the  community  seems  to  be  on  the  right  track.  Although  progress  in 
modelling  human  factors  has  been  slow  over  the  past  decade,  other  forums  have  been  reporting  a  number 
of  theoretical  and  applied  papers  on  human  behaviour  and  performance  modelling.  While  we  are  still  a 
long  way  from  turnkey  models  of  operators,  the  consensus  that  various  modelling  approaches  are  useful 
gives  confidence  to  move  ahead  in  the  field  from  a  variety  of  perspectives. 

Some  of  the  key  recommendations  to  NATO  are: 

1)  Foster  closer  ties  among  NATO  M&S  stakeholders,  including  military,  analysts  (SAS)  and  human 
sciences  (HFM)  specialists  to  ensure  that  appropriate  models  are  being  used  to  represent  the 
human  element  of  military  simulations,  or  if  this  is  not  practicable,  then  to  recognize  the 
limitations  of  the  models  being  used. 

2)  Establish  a  mechanism  to  collect  and  disseminate  operational  or  training  data  that  are  suitable  for 
developing  and  validating  models  of  individual  and  group  behaviour  and  performance,  particularly 
data  that  supports  modelling  for  the  “3-Block  War”  concept  of  operation. 

3)  Promote  the  development  of  an  open  architecture  or  interface  specification  that  supports 
interaction  of  operator  models  from  a  variety  of  sources  within  military  synthetic  environments, 
particularly  those  environments  that  deal  with  the  broader  issues  of  Effects  Based  Operations  and 
the  activities  characterized  as  a  3-Block  War. 

4)  Promote  the  development  and  publication  of  formal  models  that  support  the  analysis  of  Effects 
Based  Operations,  including  the  influences  of  culture,  motivation  and  public  opinion  on  individual 
and  group  behaviour. 

5)  Promote  the  use  of  HBR  modelling  approaches  in  military  M&S  for  which  the  actions  of  individuals 
and  teams  play  a  critical  role  in  the  observed  behaviours  and  outcomes. 
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Representation  du  comportement  humain 
dans  des  moderations  creatives 
(RTO-MP-HFM- 143) 


Synthese 

La  RTO  de  l’OTAN  a  coordonne  un  certain  nombre  d’ etudes  sur  la  moderation  du  personnel  et  des 
moyens  pour  les  simulations  militaires  (Dompke  &  Scheckeler,  1999  ;  RTO,  2001)  et  l’interet  concernant 
la  modelisation  des  hommes  se  poursuit  dans  un  certain  nombre  de  panels  RTO  de  l’OTAN.  La  RTO 
HFM  RTG-128  sur  la  representation  du  comportement  humain  (HBR)  dans  des  moderations  creatives 
fournit  des  conseils  aux  analystes  operationnels  et  aux  ingenieurs  specialises  sur  les  facteurs  humains  qui 
peuvent  etre  inseres  dans  des  modeles  operationnels.  De  tels  modeles  sont  utilises  pour  differentes 
evaluations  -  notamment  pour  les  materiels,  le  soutien  en  mission  et  la  tactique  -  ainsi  que  pour  la  formation. 
Le  but  du  HFM  RTG-128  est  de  faire  des  recommandations  a  la  communaute  de  moderation  HBR 
concernant  l’utilisation  des  facteurs  humains  dans  des  modeles  constructifs  d’une  maniere  methodique, 
en  preparant  le  terrain  par  une  standardisation  et  une  reutilisation  de  modules  decrivant  des  facteurs  humains 
specifiques. 

Le  HFM-143/RSM  « Representation  du  comportement  humain  dans  des  moderations  creatives » 
a  ete  amorce  au  sein  du  RTG-128  comme  partie  integrante  de  ses  activites  en  vue  de  faire  progresser  le 
domaine  de  la  moderation  humaine  pour  le  M&S  de  l’OTAN.  Trente-trois  personnes  s’interessant  a  la 
moderation  du  comportement  humain  et  aux  performances  humaines  provenant  de  sept  membres  de 
l’OTAN  et  de  pays  Partenaires  pour  la  Paix  se  sont  done  reunis  au  Canadian  Forces  College  a  Toronto  au 
Canada  pour  un  atelier  de  2  jours  (30-31  mai  2007)  pour  discuter,  debattre  et  echanger  des  idees  sur  les 
aspects  de  la  moderation  humaine. 

Chaque  jour  a  debute  par  une  presentation  sur  le  role  de  l’HBR  dans  la  simulation.  La  premiere  presentation 
faite  par  le  Dr.  Robert  Foster,  de  DDR&E,  a  donne  le  point  de  vue  des  gouvernements  et  de  l’OTAN, 
en  soulignant  le  besoin  d’aller  au-dela  des  exigences  concernant  la  moderation  de  combat  traditionnelle 
pour  inclure  le  domaine  plus  large  des  activites  militaires  sur  l’aide  humanitaire  et  la  reconstruction  d’un 
pays,  en  se  focalisant  pour  une  bonne  part  sur  les  aspects  humains  des  operations  basees  sur  les  effets  et  des 
actions  de  combats  non  cinetiques.  La  seconde  presentation  faite  par  Mr.  Mike  Greenley,  de  CAE  Inc., 
a  donne  le  point  de  vue  industriel,  en  indiquant  le  besoin  pour  une  approche  des  bons  usages  de  la 
moderation  humaine  qui  puisse  se  developper  naturellement  plutot  que  par  contrainte.  Mais  si  l’industrie 
decouvre  un  marche  prometteur,  comme  la  formation  plus  large  du  personnel,  les  societes  participeront  alors 
a  cette  evolution  en  tant  que  partenaires.  Aussi,  les  sept  domaines  d’interet  suivants  ont  ete  debattus  dans  les 
sessions  suivantes  : 

1)  Quels  facteurs  humains  sont  concernes  dans  une  operation  ? 

2)  Representation  de  la  tache  humaine  en  M&S. 

3)  Generation  du  comportement  -  variability  et  choix. 

4)  Le  concept  des  moderateurs. 

5)  Mesures  mentales  militairement  pertinentes  obtenues  -  charge  de  travail,  evaluation  de  la  situation 
et  autres  concepts  utiles. 
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6)  Complexity,  hierarchie,  modularity  et  validity  dans  les  architectures  HBR. 

7)  Du  comportement  individuel  au  comportement  collectif. 

Chaque  session  comprenait  des  remarques  d’ introduction  par  le  president  de  session  et  des  presentations 
breves  par  des  chercheurs  dans  le  domaine  pour  ouvrir  les  perspectives  sur  les  sujets  a  debattre.  Cela  etait 
suivi  d’un  debat  general  parmi  les  participants.  Le  principe  d’ etude  de  point  particulier  en  session  pleniere 
au  travers  de  presentations  courtes  a,  semble-t-il,  ete  bien  accepte  et  dans  les  debats,  une  participation 
etendue  de  tous  les  participants  a  ete  constatee. 

II  y  a  eu  un  consensus  general  sur  les  points  de  vue  de  l’approche  HFM-128  sur  la  fagon  de  renseigner  les 
facteurs  humains  dans  le  cadre  de  l’HBR  et  sur  la  bonne  direction  que  la  communaute  semblait  prendre. 
Bien  que  les  progres  sur  les  facteurs  humains  aient  ete  lents  ces  dix  dernieres  annees,  d’autres  forums  ont 
rendu  compte  d’un  certain  nombre  de  communications  theoriques  et  appliquees  sur  la  modelisation  du 
comportement  humain  et  des  performances  humaines.  Bien  que  nous  soyons  encore  loin  de  modeles  et 
d’operateurs  cles  en  main,  le  consensus  sur  l’utilite  de  differentes  approches  de  modelisation  donne 
l’assurance  qu’il  faut  poursuivre  dans  le  domaine  a  partir  de  differentes  perspectives. 

Quelques  recommandations  cles  ont  ete  donnees  a  l’OTAN  : 

1)  Renforcer  les  liens  entre  les  intervenants  M&S  de  l’OTAN,  en  incluant  les  specialistes  militaires, 
les  analystes  (SAS)  et  les  specialistes  des  sciences  humaines  (HFM)  pour  s’assurer  que  les  modeles 
appropries  ont  ete  utilises  pour  representer  1’  element  humain  dans  les  simulations  militaires,  ou  si  ce 
n’est  pas  possible,  reconnaitre  alors  les  limitations  des  modeles  utilises. 

2)  Etablir  un  mecanisme  de  recuperation  et  de  diffusion  des  donnees  operationnelles  ou  d’entramement 
formation  qui  soient  adaptes  pour  developper  ou  valider  des  modeles  de  comportement  et  de 
performance  individuels  et  collectifs,  en  particulier  des  donnees  qui  soutiennent  la  modelisation  pour 
le  concept  d’ operation  «  3-Block  War  ». 

3)  Promouvoir  le  developpement  d’une  architecture  ouverte  ou  d’une  specification  d’interface  qui 
soutienne  V  interaction  des  modeles  d’operateur  a  partir  de  differentes  sources  dans  les 
environnements  synthetiques  militaires  en  particulier  ceux  qui  traitent  des  reponses  plus  larges  des 
operations  en  fonction  des  effets  et  des  activites  considerees  comme  «  3-Block  War  ». 

4)  Promouvoir  le  developpement  et  la  publication  de  modeles  formels  qui  soutiennent  V  analyse  des 
operations  basees  sur  les  effets,  en  incluant  les  influences  culturelles,  la  motivation  et  V  opinion 
publique  sur  le  comportement  individuel  et  collectif. 

5)  Promouvoir  V utilisation  des  approches  de  modelisation  HBR  dans  la  M&S  militaire  pour  laquelle 
les  actions  individuelles  et  collectives  jouent  un  role  critique  dans  les  comportements  observes  et 
les  resultats. 
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NATO  RTO  has  coordinated  a  number  of  studies  into  the  modelling  of  personnel  and  platforms  for  use  in 
military  simulations  (Dompke  &  Scheckeler,  1999;  RTO,  2001)  and  interest  in  human  modelling  continues  in 
a  number  of  NATO  RTO  panels.  Pew  and  Mayor’s  (1998)  review  of  the  field  identified  a  number  of  shortfalls 
in  the  current  state  of  the  art,  as  have  the  above  mentioned  NATO  and  TTCP  (The  Technical  Cooperation 
Program,  Hawkins  et  al.,  2003)  reports.  The  RTO  HFM-128/RTG  on  Human  Behaviour  Representation  in 
Constructive  Simulation  is  in  the  process  of  providing  guidance  for  operational  analysts,  modellers  and 
engineers  on  human  factors  that  can  be  included  in  operational  models  and  simulations  that  have  personnel  as 
a  core  element.  Such  models  are  used  for  various  assessments  -  notably  materiel,  mission  support  and  tactics 
-  as  well  as  training. 

The  goal  of  HFM-128/RTG  is  to  make  recommendations  to  the  HBR  modelling  community  regarding  the  use  of 
human  factors  in  constructive  models  in  a  systematic  way,  paving  the  way  for  standardisation  and  re-use  of 
modules  describing  specific  human  factors.  The  modeller  would  then  plug-and-play  modules  within  standardised 
human  factors  architecture.  HFM-128/RTG  identified  the  requirement  to  hold  a  Specialists’  Meeting  in  which 
the  approach  that  the  panel  is  following  could  be  communicated  to  and  challenged  by  specialists  in  the  field,  as 
well  as  to  discuss  a  number  of  issues  relevant  to  human  modelling  for  military  applications. 

HFM-143/RSM  “Human  Behaviour  Representation  in  Constructive  Modelling”  was  initiated  by  HFM-128/ 
RTG  as  part  of  its  activities  to  further  the  field  of  human  modelling  for  NATO  M&S.  A  RTO  Specialists’ 
Meeting  (RSM)  technical  team  activity  aims  at  promoting  exchange  of  state-of-the-art  knowledge  among  an 
audience  of  specialists  with  selected  speakers  on  an  important  scientific  or  applied  topic.  The  prime  purpose 
of  an  RSM  is  to  enhance  the  capability  of  the  NATO  R&D  community  to  respond  adequately  to  the  military 
requirements  of  NATO. 

Thirty-three  people  with  an  interest  in  modelling  human  behaviour  and  performance  from  seven  NATO 
member  and  Partners-for-Peace  countries  gathered  at  the  Canadian  Forces  College  in  Toronto,  Canada  for  a 
2-day  workshop  (30-31  May  2007)  to  discuss,  debate  and  exchange  ideas  on  aspects  of  human  modelling. 
The  list  of  attendees  is  included  in  the  Appendices.  The  list  of  participants  and  the  meeting  agenda  are 
provided  in  the  Appendices  1  and  2  respectively. 

The  RSM  was  opened  by  the  Co-chair  Mr.  Cain,  stressing  that  the  format  of  the  meeting  was  intended  to 
promote  discussion  rather  than  listening  to  formal  presentations  and  that  participation  in  the  discussions  by 
attendees  was  essential  to  the  meeting’s  success.  Two  keynote  speakers  presented  perspectives  from  NATO  as 
well  as  from  industry.  These  presentations  served  to  set  the  stage  for  discussions,  focussing  on  the  HBR 
requirements  and  technologies  that  could  be  brought  to  bear  on  NATO  M&S  for  the  representation  of 
personnel. 

The  following  seven  topic  areas  were  discussed  in  separate  sessions  at  the  Specialists’  Meeting  and  these 
proceedings  summarize  those  discussions: 

1)  What  Human  Factors  does  an  operation  involve? 

2)  Human  task  representation  in  M&S. 

3)  Behaviour  generation  -  variability  and  choice. 
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4)  The  concept  of  moderators. 

5)  Militarily  relevant  mental  output  measures  -  workload,  situation  awareness  and  other  useful  concepts. 

6)  Complexity,  Hierarchy,  Modularity,  and  Validity  in  HBR  Architectures. 

7)  From  individual  to  group  behaviour. 

Each  session  comprised  some  introductory  remarks  by  session  chairs  and  brief  presentations  by  researchers  in 
the  field  to  offer  perspectives  on  the  discussion  topic.  This  was  followed  by  a  general  discussion  among  the 
participants.  Presentations  were  kept  brief  to  provide  time  to  explore  and  debate  ideas.  This  format  seemed  to  be 
well  received  and  prompted  widespread  participation  by  the  attendees. 

Summaries  of  the  session  discussions  were  written  by  the  session  chairs  with  input  from  the  panel  and  various 
presenters;  slides  from  the  presentations  are  included  in  Appendix  1.  The  SM  technical  evaluator,  Mr.  Fraser, 
summarized  the  discussions  at  the  end  of  each  day,  bringing  out  observations  relevant  to  NATO.  Mr.  Fraser’s 
technical  evaluation  of  the  meeting  is  included  in  Chapter  12. 
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Chapter  2  -  OPENING  ADDRESS 

Dr.  Wouter  Lotens 

TNO  Human  Factors 
Kampweg  5,  PO  Box  23 
3769  ZG  Soesterberg 
THE  NETHERLANDS 


This  Specialists’  Meeting  is  the  result  of  discussions  that  were  held  over  three  years  in  the  HFM-128  task 
group  on  HBR  in  constructive  simulation.  The  similarity  in  the  names  is  not  coincidental.  HFM-128  was 
tasked  to  find  ways  to  implement  human  factors  in  operational  modelling  to  address  deficiencies  in  the  more 
typical,  mechanistic  approach  to  modelling  the  activities  of  personnel  or  platforms  controlled  by  personnel. 
The  ideas  emerging  from  that  work  will  be  documented  in  a  report  later  this  year.  However,  facing  the 
moment  that  we  have  to  formulate  recommendations  to  NATO,  HFM-128  considered  that  it  would  be  prudent 
to  consult  the  community  on  some  of  the  issues  being  reviewed.  Far  more  was  addressed  than  we  can  discuss 
here,  but  we  prepared  seven  items  that  may  receive  ample  interest  and  are  at  the  core  of  behaviour  generation 
in  models.  The  target  audience  for  this  Specialists’  Meeting  includes  human  factors  specialists,  developers  of 
modelling  systems  and  potential  users.  It  was  observed  that  there  are  few  forums  for  these  stakeholders  to 
meet  collectively  and  that  having  them  together  would  be  of  benefit  to  all  parties. 

HFM-128  concluded  that  several  deficiencies  in  human  behaviour  modelling  still  exist.  Previous  reports,  such 
as  from  SAS-017  (RTO,  2001)  and  the  two  authoritative  books  by  Pew  and  Mavor  (1998)  and  Gluck  and  Pew 
(2005),  identified  a  number  of  outstanding  questions  including: 

•  How  can  a  balanced  human  factors  input  be  given  to  operational  modelling  involving  the  relevant 
human  factors  with  the  appropriate  accuracy? 

•  How  can  a  range  of  likely  behaviours  be  generated  in  a  given  situation  and  how  is  the  selection  made 
for  the  single  behaviour  that  is  activated? 

•  How  do  we  obtain  models  that  are  fit  for  purpose,  rather  than  selecting  the  nearest  available  model? 

•  What  performance  metrics  can  be  applied  in  the  military  context  of  the  simulated  problem? 

•  Can  we  limit  the  effort  of  a  study  by  reuse  of  sub-models  and  reduction  of  complexity? 

HFM-128  developed  a  vision  on  these  challenges  based  on  scientific  insights  and  operational  developments  that 
will  be  reported  in  its  final  report.  Typically,  in  many  applied  models  and  architectures,  scientific  knowledge  is 
neglected  in  the  favour  of  convenient  engineering  techniques  that  are  assumed  to  deliver  valid  results.  In  many 
instances  this  approach  is  no  more  than  an  engineering  representation  that  fills  a  similar  role  to  the  human 
functions  without  consideration  of  the  capabilities  and  limitations  of  the  personnel  they  are  attempting  to 
characterize.  The  military  modelling  and  analysis  community  need  solid  science-based  representations  of  these 
human  functions  to  have  confidence  in  the  validity  of  the  resulting  performance. 

Studies  need  to  be  carefully  framed  to  provide  the  context  and  military  perspective  required  to  produce  useful 
results.  Certainly,  the  advent  of  effects  based  operations  has  a  profound  impact  on  what  behaviours  are 
demonstrated,  the  performance  that  is  realized  and  the  metrics  that  should  be  applied  to  assess  the  outcomes. 
Classical  measures  such  as  time  to  completion  and  casualty  counts  may  be  subordinate  to  operational  success 
or  failure. 
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One  way  to  reduce  modelling  complexity  is  through  a  moderator  concept  that  allows  analysts  to  separate 
human  functions  from  states  and  traits  yet  still  incorporate  individual  differences  to  affect  performance  on 
subtasks.  These  moderators  affect  task  performance  differentially,  depending  on  the  nature  of  the  task  and  the 
moderator.  The  appropriateness  of  the  moderator  concept  in  human  operator  model  is  under  debate  but  it  has 
been  successfully  applied  in  some  instances. 

Cognition  has  a  profound  effect  on  behaviour  and,  in  particular,  decision  making.  Behaviour  representation 
generally  implies  that  a  number  of  response  options  are  available  for  given  situations  and  that  choices  are  not 
scripted,  resulting  in  a  different  course  of  action.  The  control  of  the  decision  making  and  the  cognitive  process 
involved  in  decision  making  are  key  to  generating  plausible,  validated  behaviour  and  performance.  Few  if  any 
of  the  in-service  M&S  tools  consider  aspects  such  as  expertise  and  Naturalistic  Decision  Making,  resorting 
instead  to  more  mechanistic,  Artificial  Intelligence  approaches. 

This  is  a  potentially  powerful  vision  on  a  difficult  subject  and  there  are  no  ready  answers.  The  discussions  in 
this  Specialists’  Meeting  have  to  reveal  if  the  community  agrees  and  NATO  can  be  recommended  to  follow 
this  route. 
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Chapter  3  -  KEYNOTE  ADDRESS:  DAY  1  -  REPRESENTATION  OF 
HUMAN  BEHAVIOUR  IN  CONSTRUCTIVE  SIMULATION 

Session  Chair:  Dr.  Wouter  Lotens 

TNO  Human  Factors 
Kampweg  5,  PO  Box  23 
3769  ZG  Soesterberg 
THE  NETHERLANDS 

Presenter:  Dr.  Robert  E.  Foster 

Director  BioSystems 

Office  of  the  Director,  Defense  Research  and  Engineering 
1777  North  Kent  Street,  Suite  9030 
Arlington,  VA  22209 
USA 


Dr.  Foster  set  the  framework  for  considering  HBR  “futures”  by  defining  the  “domains”  for  military 
applications  of  HBR  in  models  and  simulations.  The  key  point  was  that  developments  in  HBR  to  date  have 
concentrated  on  tactical-conventional  warfare  and  the  emergence  of  world-wide  “irregular  warfare”  and 
“small  wars”  drive  the  present  and  future  need  for  a  new  scope  of  modelling.  This  emergent  future  was 
summarized  in  the  domain  space  graph  repeated  in  Figure  3-1.  He  pointed  out  that  the  new  challenges  to  HBR 
in  simulations  are  emerging  from  current  military  missions  and  can  be  expressed  in  the  following  shifts  in  the 
demand  for  research:  from  a  conventional  warfare  tactical  focus  (the  “B”  space  in  the  graph)  to  tactical  - 
operational  -  strategic  levels  of  military  operations  in  irregular  conflict  environments  (a  HBR  project  depicted 
as  “A”  in  the  graph);  from  weapons-centered  conventional  missions  to  missions  involving  stability,  support, 
reconstruction  (SSTR)  and  governance  support;  from  24  hour  tactical  operations  to  longer  term  strategic 
initiatives  lasting  years.  The  new  context  is  ‘Non-kinetic  warfare’  in  contrast  to  ‘Military  formation-based 
warfare’  and  the  military  focus  is  the  spectrum  of  operations  from  irregular  warfare  to  post  event 
reconstruction.  Non-kinetic  warfare  may  require  new  approaches  to  modelling,  whereas  NATO  militaries 
know  how  to  develop  and  use  “combat  simulation”  for  analysis  of  formation  based  warfare.  These  existing 
real-time,  integrated  synthetic  environments  do  not  appear  to  be  capable  of  supporting  non-kinetic  warfare 
modelling.  There  is  a  need  for  integrating  available  social  science  knowledge  and  models,  possibly  into  new 
modeling  environments  such  as  “agent-based  models”.  This  is  due  to  the  possibility  that  extant  military 
constructive  simulations  may  not  be  able  to  be  engineered  to  address  social  and  cultural  aspects  of  HBR  as 
applied  to  non-kinetic  military  problems.  While  the  US  DoD  has  an  emerging  investment  in  research  on 
human  terrain  and  the  socio-cultural  understanding  of  conflict  environments,  much  work  in  the  M&S  area 
needs  to  be  undertaken  to  bring  this  science  into  the  practice  of  formal  military  constructive  simulations. 
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Figure  3-1 :  Conceptual  Relationships  Among  the  Factors  Relevant  to  Traditional  Military  Manoeuvre 
Warfare  Simulations  and  the  Evolving  Focus  of  Strategic  Modelling  for  Non-Kinetic  Warfare. 


Among  the  other  needs  of  NATO  is  authoring  technology  for  developing  human  operator  models.  Operational 
experience  has  to  flow  into  simulation  environments  more  directly  and  the  ideal  way  would  be  to  let  military 
Operational  Analysts  do  the  modelling,  rather  than  the  modeller.  Later  in  this  meeting  it  was  doubted  if  this 
can  be  achieved  anytime  soon,  because  of  the  complexity  and  limitations  of  the  human  models,  yet  Pew  and 
Mavor  (1998)  acknowledged  the  need  for  composable  behaviours  almost  a  decade  ago. 

A  third  need  is  validation,  particularly  of  higher  level  simulations.  Lack  of  data  is  an  obstacle  to  validation  of 
practical  military  simulations.  The  use  of  moderators  is  regarded  as  a  primitive  way  to  change  performance. 
Confusion  may  arise  about  what  a  moderator  is.  Here,  it  was  interpreted  as  a  “dial”  that  influences  the  entity 
behaviour  in  a  generic  way,  without  distinction  of  goals  and  tasks,  or  as  “white  cell”  entity  handlers  who 
manually  control  the  behaviour  of  SAFs.  Later  in  the  meeting,  a  different  definition  of  moderators  was  used 
that  refers  to  formal,  scientifically  sound  cognitive  or  physiological  models  that  affect  performance, 
for  instance  accurate  thermal  physiological,  physical  fatigue,  or  mental  workload  models.  Game  environments 
have  the  potential  to  be  used  as  a  test  environment,  notably  “Real  World”  (DARPA)  and  “A  Force  More 
Powerful”  (http://www.aforcemorepowerful.org/game/index.php).  But,  Dr.  Foster  pointed  out,  with  the  lack  of 
evidence  of  validation  or  inclusion  of  scientifically  based  human  moderating  functions,  do  we  have 
confidence  in  the  predictions  from  these  simulations?  What  should  the  research  programs  of  the  NATO 
nations  address  to  move  validation  forward? 

The  audience  wanted  to  know  how  scientists  can  collect  the  data  required  for  model  development  and 
validation  from  the  field.  Although  direct  observation  would  be  difficult,  web  based  data  collection  using  tools 
are  in  existence  (e.g.,  LexisNexis;  http://www.lexisnexis.com/)  and  could  be  assessed  for  feasibility.  Another 
option  is  to  leverage  “blog”,  “chat  room”  or  other  social  network  technology.  The  US  Army  is  exploring  this 
opportunity  for  data  sharing  by  operating  a  ‘soldier- team-commander’  website  to  gather  such  data  and 
lessons-learned  from  individuals  in-theatre.  Hypotheses  might  be  tested  on  these  raw  observations  to  develop 
more  general  conclusions,  as  this  is  standard  procedure  in  marketing  research. 
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Dr.  Foster  admitted  that  modelling  traditional/conventional  combat  activities  is  still  prevalent.  He  was  hopeful 
for  progress  in  that  there  is  expanding  interest  in  the  non-traditional  modelling:  in  2006  some  56  projects  were 
sponsored  on  socio-cultural  modelling  that  could  be  applied  to  analysis  of  Effects  Based  Operations  in 
irregular  warfare  scenarios.  And,  there  is  an  increasing  understanding  between  engineers,  computer  scientists 
and  the  human  science  communities  that  interdisciplinary  research  is  required  to  solve  today’s  problems,  such 
as  improving  the  efficiency  and  effectiveness  of  the  various,  dispersed  humans  operating  within  Network 
Enabled  Command  and  Control. 
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Session  Chair:  Mr.  Martin  Castor 
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Presenter:  Dr.  Andy  Belyavin 

Centre  for  Human  Sciences 
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4.1  INTRODUCTION 

Pew  and  Mavor  (1998)  pointed  out  that  Human  Behaviour  Representation  (HBR)  is  a  very  challenging 
endeavour.  We  are  trying  to  understand  humans  in  complex  contexts  to  the  extent  that  we  can  write  formal 
rules  that  describe  human  thinking,  their  behaviours,  and  the  resulting  performance. 

A  model,  by  definition,  conceptual  or  computational,  is  a  simplification  of  reality.  Thus,  a  lot  of  abstractions 
and  tradeoffs  concerning  level  of  representation  have  to  be  made  in  any  modelling  project.  It  is  likely  the  case 
that  HBR  entails  more  abstractions  and  approximations  than  a  model  of  a  technical  system,  as  human  activity 
is  so  complex,  and  much  has  to  be  inferred  about  how  the  human  “works”. 

As  the  choice  of  factors  to  include  frames  any  modelling  project,  the  organizing  panel  thought  it  would  be 
fruitful  to  discuss  whether  or  not  the  choice  of  which  human  factors  to  include  in  a  specific  model  should  be 
left  up  to  the  good  judgement  of  the  modeller.  Further,  would  it  be  beneficial  if  we  could  gather  some 
recommendations  on  how  to  move  this  modelling  issue  from  art  to  more  of  a  formal  method.  Thus,  the  topic 
for  discussion  was  whether  there  are  any  formal  approaches  to  decide  what  human  factors  to  include  in 
specific  HBR  projects. 

One  answer  to  the  question  of  what  human  factors  does  an  operation  involve,  and  thus  factors  to  include  in  a 
model,  is  that  “it  depends”.  The  intention  of  the  topical  discussion  was  not  to  identify  a  long  list  of  human 
factors  that  might  affect  an  operation,  but  rather  to  see  if  good  advice  on  how  to  formally  approach  the  question 
of  what  human  factors  to  include  and  scenarios  to  use  could  be  gathered  from  the  present  HBR  experts. 

Related  questions  also  raised  at  the  introduction  were: 

•  How  do  you  link  customer  higher  level  questions  to  requirements  for  which  human  factors  to  include  in 
the  model?  (e.g.,  How  can  the  sustainability  of  a  force  be  increased?) 

•  How  to  choose  the  appropriate  fidelity  level  for  representation? 

•  How  to  do  trade-offs  between  level  of  detail  and  budget? 
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4.2  DISCUSSION 

During  the  discussion,  several  modellers  acknowledged  that  transforming  the  customer’s  higher  level  questions 
to  design  decisions  of  which  human  factors  to  include  in  a  model  was  a  challenging  problem.  To  move  this 
process  from  an  art  to  more  of  a  formal  process,  discipline  by  both  analysts  and  customers  will  need  to  be  high. 
Customers  should  clearly  define  what  they  want  and  not  move  their  objectives  as  the  study  proceeds.  Modellers 
should  be  very  clear  on  why  they  include  or  exclude  a  certain  human  factor.  A  badly  defined  study  will  likely 
result  in  invalid,  or  worse  misleading,  results  and  customers  should  be  educated  about  the  need  for  and  cost  of 
validation.  Validity  is  a  big  concern  in  all  modelling  projects.  According  to  some  modellers  present,  60  -  70%  of 
the  study  cost  may  be  spent  on  data  collection  used  to  create  and  validate  a  model.  Specialists  may  not  agree 
with  the  customer  on  what  he  may  get  and  the  customer  should  take  responsibility  for  the  basic  research  that 
needs  to  be  done  in  order  to  develop  valid  models  in  the  direction  they  need.  Data  sets  independent  from  the  data 
that  were  used  to  shape  the  model  are  needed  for  evaluation  of  predictive  validity. 

The  use  of  a  human  factors  checklist  and  rationale  for  inclusion  versus  exclusion  of  each  human  factor  in  a 
modelling  project  was  proposed.  This  checklist  could  take  the  form  of  a  guided  interview  that  could  also  be  a 
basis  for  classification  of  models.  Such  a  guided  interview  could  form  the  basis  for  discussion  with  the 
customers  on  what  to  include  and  what  to  leave  out. 

The  question  was  raised  whether  the  granularity  in  model  studies  is  sufficient  to  test  hypotheses. 
One  suggestion  was  to  analyze  the  relationship  in  the  literature  between  human  factors  and  state  variables 
to  deduce  specific  hypotheses,  rather  than  doing  statistical  analyses.  Statistics  do  not  reveal  causes,  just 
co-occurrence. 

When  a  model  permits  the  manipulation  of  many  variables,  it  is  good  practice  to  initially  manipulate  only  one 
at  a  time  to  gain  understanding  of  how  this  factor  affects  the  model.  Even  though  this  might  be  part  of  basic 
good  practice  for  any  scientific  study,  at  least  one  anecdote  from  the  discussion  suggests  that  some  end  users 
of  modelling  software  (rather  than  modelling  specialists)  manipulate  many  or  all  variables  at  once,  stressing 
the  models  beyond  their  original  design  scope  and  causing  them  to  fail  in  unexpected  ways  during  execution. 
Development  and  use  of  models  still  requires  in-depth  knowledge  of  the  application  domain,  the  model, 
the  modelling  environment,  the  simulation  settings,  and  the  objectives  of  the  study  to  have  a  reasonable 
expectation  that  the  results  obtained  are  meaningful. 

The  decision  of  which  human  factors  to  include  in  a  model  should  be  taken  in  light  of  where  the  model  will  be 
used  (e.g.,  an  acquisition  project).  If  the  factor,  due  to  what  stage  the  project  is  in  or  other  reasons,  cannot  be 
affected,  the  rationale  for  inclusion  of  this  factor  decreases,  i.e.,  the  “So  what?”  question.  For  example, 
if  recruitment  to  the  army  will  never  be  based  on  personality  type,  the  inclusion  of  this  factor  in  a  model  of 
personnel  selection  and  placement  procedures  might  be  less  important  than  some  other  factor  that  can  be 
affected. 

The  short-term  versus  the  long-term  requirements  of  a  model  can  provide  input  to  which  human  factors  to 
include  as  well  as  the  appropriate  level  of  representation  and  abstraction.  A  simple  model  can  be  cheaper 
initially,  but  can  become  more  expensive  in  the  end  if  it  has  to  be  remodelled  rather  than  simply  elaborated  in 
the  future  projects.  We  must  also  realize  that  we  are  a  long  way  from  turnkey  applications  of  HBR  and  that 
there  is  no  cookbook  for  abstraction;  abstraction  and  modelling  is  an  art  and  a  skill  guided  by  knowledge. 
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4.3  SESSION  RECOMMENDATIONS 

1)  The  use  of  a  NATO  developed  checklist  of  existing  human  factors  and  presentation  of  why  each  factors 
was  included  or  excluded  in  a  specific  model  should  be  part  of  good  practice. 

2)  It  is  still  necessary  to  emphasize  the  need  for  “hooks”  in  the  simulation  of  the  physical  environment  where 
the  behaviour  models  can  get  information  form.  It  is  also  necessary  to  link  HBRs  to  a  context  so  that  the 
effects  of  model  behaviour  are  seen  more  clearly. 

3)  NATO  should  develop  an  ongoing  capability  to  collect  and  disseminate  data  to  support  modelling  of  non- 
kinetic  warfare,  asymmetric  situations  and  the  long  terms  problems  faced  by  commanders  on  a  strategic 
level,  as  addressed  in  Bob  Foster’s  keynote  presentation.  Much  of  this  data  will  involve  human  factors 
from  a  number  of  domains. 


4.4  PRESENTATION 

In  order  to  provide  one  example  of  a  modelling  project  where  the  discussion  topic  could  be  exemplified  Andy 
Belyavin  presented  an  overview  of  a  project  concerning  the  prediction  of  aircraft  collisions  in  the  UK  Low 
Flight  System.  The  example  showed  that  considerable  insight  into  problems  can  be  achieved  by  including 
relatively  few  human  factors,  but  that  these  factors  have  to  be  chosen  judiciously  to  be  successful. 

Also  discussed  was  the  need  for  a  modular  modelling  environment  that  allows  the  analyst  to  expand  the  model 
as  additional  questions  are  asked  that  were  not  part  of  the  original  project  problem  scope  or  design  specification. 


RTO-MP-HFM-1 43 


4-3 


WHAT  HUMAN  FACTORS  DOES  THE  OPERATION  INVOLVE? 


4-4 


RTO-MP-HFM-1 43 


ORGANIZATION 


Chapter  5  -  HUMAN  TASK  REPRESENTATION  IN  M&S 

Session  Chair:  Dr.  Laurel  Allender 

Army  Research  Laboratory 
Human  Research  &  Engineering  Directorate 
Attn:  AMSRD-ARL-HR-SE 
Aberdeen  Proving  Ground,  MD  21005-5425 
USA 

Presenter:  Dr.  Oshin  Vartanian 

Defence  Research  and  Development  Canada 
1133  Sheppard  Avenue  West 
Toronto,  Ontario  M3M  3B9 
CANADA 


5.1  BACKGROUND 

The  representation  of  human  behaviour  in  M&S  begins  with  the  question  of  how  to  best  represent  the  tasks 
being  performed,  that  is,  the  activities  being  carried  out  that  constitute  behaviour.  How  should  the  steps  and 
sub-steps  of  activity,  the  overt  and  the  ‘black  box’  aspects  of  a  task,  the  motor  and  the  cognitive  decision 
making  elements  of  a  task  be  translated  into  M&S  software?  How  should  you  begin?  Given  that  most  of  the 
models  of  human  behaviour  in  use  today  have  roots  in  the  human  factors  community,  task  representation 
typically  begins  with  some  form  of  task  analysis.  Subject  matter  experts  (SMEs)  are  used  to  develop  a 
structured  ‘list’  that  captures  the  tasks  of  interest.  Task  analysis  methods  vary  with  respect  to  the  details 
depending  on  whether  it  is  a  mission-function-task  decomposition,  a  cognitive  task  analysis,  a  goal-directed 
task  analysis,  a  work-centred  approach,  etc.,  but  the  general  result  the  same.  Generally,  tasks  are  modelled  as  a 
sequential  or  networked  series  of  steps  of  behaviour  on  the  order  of  seconds  or  minutes.  For  many  purposes, 
when  the  tasks  to  be  represented  are  orderly  and  operators  follow  procedures,  this  may  suffice;  however,  there 
are  several  factors  that  may  influence  human  task  representation  in  M&S  and  humans  often  deviate  from 
procedures,  often  with  good  reason. 


5.2  DISCUSSION 

5.2.1  Expertise  and  Learning 

Two  related  factors  that  influence  how  a  task  is  to  be  represented  are  expertise  and  learning.  There  is  new 
evidence  emerging  from  research  using  brain  imaging  such  as  functional  Magnetic  Resonance  Imaging 
(fMRI)  that  shows  that  experts  have  an  overall  lower  brain  activation  during  performance  of  the  tasks  of 
interest  than  do  non-experts,  suggesting  that  individual  steps  have  become  one  larger  step  or  ‘chunk’.  This 
result  is  consistent  with  the  finding  reported  in  the  naturalistic  decision  making  literature  (see  Klein  for 
additional  insight:  Zsambok  &  Klein,  1997)  that  experts  very  often  cannot  express  the  detailed  steps  they 
performed  as  a  part  of  a  larger  task.  This  is  both  problematic  and  interesting  for  HBR.  On  the  one  hand,  SMEs 
are  a  main  source  of  information  about  the  tasks  to  be  performed,  but  if  they  cannot  verbalize  what  the  steps 
or  procedures  are,  then  the  task  analysis  is  incomplete  in  some  sense.  On  the  other  hand,  the  difference 
between  novices  and  experts  (going  from  many  task  steps  to  a  single,  more  efficient  task  step)  may  be 
important  to  represent  directly.  Discussion  of  this  topic  at  the  RSM  covered  the  role  of  implicit  judgement, 
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limited  deliberation  (not  4 over- thinking’  a  decision),  recognition-primed  decision  making,  pattern-matching  in 
expert  decision  making  and  the  associated  implications  for  task  representation  in  M&S.  Most  of  the  discussion 
referenced  military  decision  making  but  pertinent  examples  from  economic  decision  making  (e.g.,  buying  a 
car)  were  given  as  well. 

In  addition  to  experts  thinking  of  the  task  in  a  unitary  fashion,  as  a  ‘chunk’,  rather  than  as  a  series  of  sub¬ 
steps,  there  is  evidence  that,  since  they  are  actually  using  fewer  cognitive  resources,  they  are  better  able  to 
‘multi-task’  and  attend  to  other  aspects  of  the  environment,  which,  in  turn  makes  them  better  able  to  detect 
anomalies  and  to  do  error  correction,  all  of  which  would  need  to  be  accounted  for  in  a  model  of  expert 
performance.  The  matter  of  pattern-matching  raised  the  question  of  whether  this  simplifies  the  modelling 
process  or  creates  new  difficulties.  A  model  that  includes  pattern-matching,  ideally,  would  include  matching 
based  not  only  on  surface  features,  but  also  based  on  temporal,  relational  and  procedural  features 
characteristic  of  analogical  reasoning  theories  (Thagard,  2005).  What  about  the  case  when  a  novel  pattern  of 
events  unfolds  that  does  not  have  a  match?  It  was  suggested  that  a  two-system  or  hybrid  model  that  includes 
both  symbolic  and  subsymbolic  processes  (such  as  ACT-R:  Anderson  &  Lebiere,  1998)  may  be  useful  for  this 
level  of  modelling. 

Modelling  expert  performance,  however,  begs  the  question  of  how  the  expertise  was  gained  in  the  first  place. 
So,  the  question  is,  could  a  model  of  expert  behaviour  be  created  by  having  the  model  learn  in  the  same 
fashion  as  a  person?  This  could  be  accomplished  by  repeated  exposures  to  the  tasks  to  be  learned,  with  a 
representation  being  built  up  and  modified  over  time.  Some  modelling  approaches  offer  the  capability  to  learn, 
although  they  are  not  in  widespread  use  as  of  yet. 

5.2.2  Context 

Another  factor  that  will  influence  task  representation  is  context  and,  specifically,  the  degree  to  which  context 
is  ‘hard-coded’  versus  one  more  element  the  HBR  must  draw  from  the  M&S  environment.  On  the  surface, 
humans  implicitly  understand  the  nature  of  a  task  without  it  being  described  in  detail,  without  it  being  broken 
down  into  sub-tasks  and  small  steps.  The  challenge  arises  when  the  task  is  to  be  represented  computationally. 
At  home,  the  task  ‘wash  the  dishes’  evokes  the  entire  context  (the  dishes  are  dirty,  that  hot  water,  a  basin, 
soap,  and  so  on)  is  available  and  the  motions  follow  as  a  matter  of  routine.  However,  language  is  ambiguous; 
different  cultures  will  have  different  interpretations  of  the  same  general  task  language.  The  humorous  military 
example  was  given  of  the  task  ‘secure  the  building’,  which  is  interpreted  differently  by  the  different  services. 

•  The  Army  would  surround  the  building  with  defensive  fortifications,  tanks  and  concertina  wire. 

•  The  Marine  Corps  would  assault  the  building,  using  overlapping  fields  of  fire  from  all  appropriate 

points  on  the  perimeter. 

•  The  Navy  would  turn  out  the  lights  and  lock  the  doors. 

•  The  Air  Force  would  take  out  a  three-year  lease  with  an  option  to  buy  the  building. 

The  RSM  discussion  covered  two  points:  One,  that  representing  context  -  including  the  immediately  prior 
history,  the  training,  social,  and  cultural  background  being  instantiated,  the  ability  of  humans  to  sort  through 
the  ambiguities  of  language  -  are  challenges  for  high  fidelity  task  representation;  Two,  that  ensuring  a 
common  understanding  of  the  nature  of  the  task  representation  between  the  SMEs,  the  customer,  the  modeller, 
and  others  in  the  modelling  community  is  a  problem  of  semantics  and  of  perspectives  that  must  be  dealt  with 
directly  at  the  beginning  of  a  modelling  effort  and  revisited  throughout  the  project  to  ensure  a  good  outcome. 
For  example,  the  definition  of  a  ‘rule’  differs  depending  on  the  modelling  approach  being  used.  A  suggestion 
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from  the  RSM  attendees  for  distinguishing  between  rules  and  pattern  matching  (which  may  be  implemented  as 
a  logical  rule  in  the  software)  was  that  perhaps  rules  can  be  expressed  explicitly  as  opposed  implicit  pattern- 
matching  that  might  be  expressed  sub- symbolically  or  via  a  neural  network. 

5.2.3  Goal-Directed  Representations 

A  third  factor  is  the  degree  to  which  tasks  should  be  represented  as  goal-directed  versus  as  a  set  of  steps  that 
are  executed  essentially  in  order  once  the  series  is  initiated.  A  goal-directed  representation  better  enables  an 
HBR  to  switch  priorities  based  on  new  or  additional  information  available  in  the  M&S.  Goal  directed 
representations  may  also  better  characterize  naturally  occurring  errors  such  as  errors  of  commission  and  to 
have  more  flexible  error  recovery,  to  perform  multi-tasking  requirements  in  a  realistic  fashion,  etc.  A  related 
aspect  of  a  goal-directed  task  representation  (such  as  Hierarchical  Task  Analysis,  HTA:  Annett,  2003;  Annett 
&  Duncan,  1967)  is  whether  the  representation  is  of  the  precise  methods  to  be  accomplished  (e.g.,  these  six 
steps  in  this  exact  order)  or  of  the  end  result  (e.g.,  ‘take  the  hill’).  The  goal-directed  approach  allows  for 
variability  in  methods  employed  to  achieve  the  end,  but  also  requires  a  more  complex  model.  It  was  asserted 
by  an  RSM  attendee  that  goal-directed  modelling  ‘does  not  come  for  free’,  as  has  been  found  with  the  BDI 
(beliefs,  desires,  intentions)  approach  to  agent  modelling. 

Additionally,  the  degree  to  which  the  methods  are  modelled  in  detail  versus  only  as  the  end  product  can  be 
thought  of  in  a  nested  fashion:  the  details  of  the  process  at  one  level  of  granularity  constitute  the  window  into 
the  black  box  of  the  higher  level,  and,  in  turn,  the  details  at  that  level  could  be  exploded  into  greater  detail  at 
the  next  level.  For  example,  the  ‘task’  of  sending  a  message  could  be  modelled  as  a  series  of  steps  (turn  on 
system,  type,  check,  press  send),  or  the  task  could  be  modelled  as  just  that,  a  single  action  that  takes  a  certain 
amount  of  time  in  the  aggregate.  Or,  if  it  fits  the  purpose  of  the  model,  the  task  of  sending  a  message  could  be 
modelled  at  the  level  of  milliseconds  and  involve  the  processes  of  perception,  recall  from  memory,  essentially 
simultaneous  motor  actions,  errors,  error  correction,  etc. 

5.2.4  Emotion 

A  final  factor  identified  by  the  RSM  attendees  as  influencing  the  nature  of  human  task  representation  in  M&S  is 
emotion.  The  literature  on  the  role  of  emotion  in  decision  making  is  of  increasing  interest  to  the  military 
modelling  community  with  new  emphasis  on  operations  other  than  war  and  effects  based  operations.  Positive 
versus  negative  emotions  seem  to  play  an  important  role  in  memory  and  decision  making.  To  some  degree, 
emotions  can  be  considered  a  stressor  or  a  performance  moderator  with  task  and  context- specific  dependencies 
indicating  a  positive  versus  a  negative  outcome. 


5.3  CONCLUSIONS 

The  choice  of  task  representation  is  influenced  by  numerous  factors  in  the  application  area  -  expertise  and 
learning,  context,  goal-directed  representation,  and  emotion.  There  are  other  factors  as  well,  many  of  which 
are  linked  to  the  other  discussion  topics  in  the  RSM:  which  human  factors  to  represent,  variability, 
moderators,  and  modelling  at  the  individual  versus  team  or  group  level.  It  can  be  concluded  that  the  outlook 
for  task  representation  is  good  for  HBR,  with  a  solid  base  having  been  established  with  a  task  network 
modelling  approach.  However,  in  order  to  represent  the  more  complex  aspects  of  human  behaviour  that 
underlie  complex  decision  making  in  the  full  range  of  military  operations,  the  additional  factors  discussed  at 
this  RSM  must  be  incorporated  into  the  HBRs  being  used  in  common  practice. 
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6.1  BACKGROUND 

Human  variability,  or  human  variation,  is  often  used  to  refer  to  the  range  of  possible  values  for  any  measurable 
human  characteristic  (mental  or  physical).  Examples  such  as  the  ability  to  learn,  differences  in  physical  strength 
or  endurance,  all  may  serve  as  moderators  to  behaviour  in  some  contexts.  Expanding  the  definition  of  human 
variability  to  include  the  range  of  possible  behavioural  outcomes,  or  behavioural  variability,  which  may  be 
observed  for  any  given  human  operating  within  an  operational  context  (e.g.,  the  representation  of  fight  versus 
flight  behaviour),  provides  a  significant  challenge  for  the  modelling  community.  In  the  context  of  improving 
realism  and  increasing  the  predictive  abilities  of  Human  Behaviour  Models  (HBMs),  developing  accurate  and 
representational  models  of  behavioural  variability  remains  a  desirable  yet  elusive  goal. 

Indeed,  the  NATO  M&S  Master  Plan  (NATO,  1998)  notes  that  current  computational  models  and  simulations 
of  operator  and  entity  behaviour  do  not  adequately  represent  human  performance,  neither  at  the  individual  nor 
the  small  group  level.  Current  Computer  Generated  Forces  (CGFs)  rely  largely  on  rule-based  behaviours  that 
are  tied  closely  to  doctrine,  and  while  doctrine  is  often  modelled,  it  is  seldom  observed  in  a  pure  form  in 
practice.  Predictability  (i.e.,  lack  of  variability),  based  on  textbook  doctrine,  may  be  desired  for  preliminary 
instruction  or  training  basic  skill  development,  but  it  is  inadequate  for  advanced  training  in  decision  making, 
situation  assessment,  experimentation  with  tactics,  or  the  evaluation  of  novel  systems  and  procedures. 

This  session  will  attempt  to  clarify  a  number  of  questions  that  remain  unanswered  with  respect  to  modelling 
behavioural  variability.  Are  there  adequate  analysis  techniques  that  can  be  leveraged  to  generate  an  adequate 
understanding  of  the  factors  underlying  behavioural  variability?  How  do  we  integrate  the  generation  of 
behavioural  alternatives  and  provide  a  mechanism  from  which  these  alternatives  are  executed?  How  does  one 
characterize  appropriate  versus  inappropriate  behaviour,  and  how  does  behaviour  relate  to  the  concept  of  error? 
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6.2  SPEAKER  PRESENTATIONS 

Mrs.  Carol  Cooper-Chapman,  Defence  Science  and  Technology  Laboratory 

Dr.  Chapman  and  colleagues  at  DSTL  are  establishing  a  modelling  framework  based  on  the  works  of  Endsley 
(Endsley,  1993,  1995,  1998),  Klein  &  Rasmussen  that  will  support  the  inclusion  of  behavioural  variability  in 
their  paradigm. 


Dr.  Emiel  Ubink,  TNO 

Dr.  Ubink  presented  an  HBR  modelling  paradigm  using  a  pandemonium  model  of  behaviour  that  TNO  is  using 
to  explore  the  performance  and  behavioural  characteristics  of  soldiers  in  operational  contexts.  The  representation 
of  behaviour  is  produced  through  the  combination  of  the  effects  of  stressors  (fatigue,  core  body  temperature, 
workload,  etc.)  with  task-specific  knowledge.  Changes  in  the  states  of  stressors  are  used  to  modify  task  selection 
thereby  producing  behavioural  variability  in  the  HBR.  In  their  current  application,  the  TNO  tool  supports  the 
representation  of  multiple  entities,  and  is  somewhat  reminiscent  of  a  SAF  system  in  that  the  level  of 
representation  appears  to  be  more  tailored  towards  modelling  the  behaviour  of  an  overall  force  structure 
(platoon,  company,  etc.)  than  individual  entities. 

The  pandemonium  model  employs  daemons  that  shriek  for  attention  (a  metaphorical  prioritisation  scheme) 
although  it  isn’t  necessarily  a  winner  take  all  rule  as  it  considers  the  resources  that  are  available  to  accomplish 
the  goal  of  the  shrieking  daemon.  The  use  of  a  resource  by  a  behaviour  leads  to  a  reduced  capacity,  as  do 
stress  and  strain,  directly  resulting  in  variable  behaviour. 


6.3  DISCUSSION 

6.3.1  Levels  of  Complexity 

Determining  the  extent  that  a  model  must  include  variability,  and  therefore  complexity,  is  an  important  aspect 
of  building  better  HBRs.  The  decision  to  include  elements  of  behavioural  variability  should  be  considered 
carefully,  and  must  be  supported  by  the  requirement  to  represent  different  aspects  of  behaviour  to  support  a 
specific  research  objective.  Caution  must  be  taken  not  to  fall  into  the  trap  of  providing  significant  levels  of 
variability  without  just-cause.  Indeed,  examples  were  provided  by  the  audience  demonstrating  that  in  extreme 
cases,  overly  complex  models  will  often  fail  to  perform  at  all,  or  exhibit  behaviours  that  are  so  unpredictable 
as  to  render  the  model  highly  unstable.  It  was  also  noted  that  the  ability  to  switch  variability  on  and  off  may 
be  an  important  feature  both  during  development  and  during  use  of  the  model. 

6.3.2  T y pes  of  V ariability 

Sources  of  behavioural  variability  typically  arise  through  two  domains:  within  individuals  or  between 
individuals.  The  ability  to  provide  an  accurate  representation  of  the  possible  actions  that  a  specific  individual 
could  use  to  respond  to  a  situation  is  important  for  individual  behavioural  variability.  This  implies  that 
modellers  must  be  aware  of  the  breadth  of  behavioural  outcomes,  and  therefore  the  task-specific  knowledge 
that  must  be  represented  within  a  model.  Data  driven  concepts  from  theorists  such  as  Klein  and  his 
Recognition  Primed  Decision  Making  (RPDM)  model  reflects  the  notion  that  variability  is  situated  in  the 
environment,  and  that  this  source  of  variability  is  what  simple  models  may  process  to  produce  more  complex 
and  variable  behaviour. 
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In  its  simplest  form,  the  representation  of  between-individual  variability  reflects  both  the  differences  in  task- 
specific  knowledge  and  drivers  of  task-selection  across  individuals.  In  the  military  domain,  obtaining  the 
relevant  task-specific  knowledge  from  Subject  Matter  Experts  (SMEs)  is  critical  to  achieving  realistic 
behavioural  variability.  However,  extending  between-individual  variability  into  the  representation  of  Cultural 
Variability  is  challenging  due  to  the  lack  of  available  SMEs  in  most  applications.  In  either  case,  the  use  of 
SME  input  to  establish  branches  for  alternate  behaviours  is  a  costly,  labour-intensive  process,  and  the  SME 
may  not  give  represented  answers  that  lead  to  plausible  behaviours. 


6.4  CONCLUSIONS 

Overall  it  was  agreed  that  variability  is  best  represented  through  between- subject  differences,  or  through 
within- subject  variability  that  reflects  both  differences  in  the  knowledge,  states  and  traits  of  an  individual. 
Current  difficulties  arise  through  the  lengthy  and  time-consuming  requirements  to  identify  relevant  knowledge 
for  given  situations,  and  establishing  realistic  links  between  state,  trait  and  environmental  characteristics  that 
give  rise  to  the  selection  of  alternative  tasks.  Future  work  should  focus  on  building  better  learning  models  that 
can  produce  new  knowledge  based  on  past  experience,  and  incorporate  this  knowledge  into  the  selection  of 
behaviours.  Ultimately  the  representation  of  variability  must  remain  plausible;  otherwise  the  inclusion  of 
variability  within  models  will  become  random  and  unexplainable. 
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Mike  Greenley  from  CAE,  Inc.  presented  a  vision  for  the  future  of  human  behaviour  modelling  from  an  industry 
perspective.  CAE  is  a  global  leader  in  the  design  of  civilian  and  military  training  systems  for  air,  land  and  sea 
applications,  having  supplied  the  defence  forces  of  more  than  50  nations  with  military  training  systems  and 
services.  CAE  offers  a  range  of  simulation  equipment  and  M&S  software  that  span  the  military  domain  into  civil 
emergency  management  applications,  where  the  incorporation  of  behaviour  models  for  entities  such  as  terrorists 
and  civil  actors  is  becoming  increasingly  important.  This  experience  provides  a  broad  base  of  M&S  expertise 
that  can  be  applied  to  predicting  the  needs  of  HBR  in  future  military  M&S. 

Mike  noted  that  accurate  representations  of  human  behaviour  are  becoming  increasingly  important  for  the 
conduct  of  military  and  civilian  M&S  activities  across  the  domains  of  training,  experimentation,  and  acquisition 
support.  However,  the  current  market  space  for  Human  Behaviour  Modelling  (HBM)  is  relatively  fragmented, 
un- standardized,  and  does  not  support  a  coherent  approach  to  the  development  of  cost-effective  HBM 
applications  that  can  adequately  address  client  requirements.  Close  collaboration  between  customer  stakeholders, 
academic  research  institutes,  and  industrial  partners  is  key  to  the  advancement  of  state-of-the-art  HBMs  and  to 
support  the  development  of  standardized  methodologies  for  the  application  of  HBM  techniques.  Successful 
collaboration  will  be  contingent  on  the  clear  identification  of  the  roles  and  responsibilities  of  industry,  academic, 
and  government  stakeholders  in  the  joint  evolution  of  HBM  technologies  and  practices. 

Mike  also  identified  that  a  requirement  for  re-usable  software  is  driven  by  the  need  within  the  client  domain  to 
reduce  the  necessity  of  re-engineering  or  re-creating  components  developed  for  one  application  into  another. 
In  the  concept  of  an  overall  simulation  architecture,  an  important  component  relevant  to  the  HBR  community 
is  to  provide  modular  and  Advanced  Intelligence  Solutions  (AIS)  characterized  by  the  figure  below,  while 
maintaining  a  balance  between  optimization  solutions  such  as  those  espoused  by  the  AI  and  Robotics 
communities,  and  the  accurate  representation  of  human  behaviour. 
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Mike  also  provided  some  insights  into  future  growth  areas  for  HBR,  including: 

1)  Significant  interest  from  the  training  community  in  providing  higher- fidelity  models  of  entity 
behaviours,  intelligent  tutors,  and  team-mates.  In  addition,  exploitation  of  the  current  training  market 
is  estimated  at  approximately  10  percent.  Expanding  this  domain  into  simulation  support  for  corporate 
leadership  is  potentially  even  larger. 

2)  Though  the  desire  for  standardized  tools  and  methodologies  for  developing  HBR  applications  is 
substantial,  standardization  will  only  succeed  where  there  is  a  clearly  identified  need  that  is  driven  by 
the  client  community  and  a  specific  application.  Orchestrated  standardization  is  extremely  difficult, 
and  history  has  demonstrated  that  this  approach  is  not  likely  to  succeed. 

3)  A  major  challenge  for  HBR  applications  is  the  difficulty  in  moving  HBR  capabilities  from  one 
simulation  domain  to  another.  However,  software  languages  are  beginning  to  emerge  for  the 
development  of  translation  languages  that  will  support  the  migration  of  modular  HBR  applications 
from  one  environment  to  another. 
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8.1  BACKGROUND 

The  interaction  between  cognition  and  context  in  the  military  environment  is  frequently  summarised  in  terms 
of  high-level  constructs  such  as  Cognitive  Workload  or  Situational  Awareness.  By  using  these  constructs  it  is 
possible  to  identify  in  qualitative  terms  the  kind  of  corrective  action  that  will  improve  crew  performance  and, 
in  some  cases,  provide  metrics  that  can  be  used  to  estimate  the  size  of  the  improvement.  The  question  arises  as 
to  whether  it  is  useful  to  construct  models  of  these  high  level  constructs  that  can  be  used  in  conjunction  with 
Human  Behaviour  Representations  to  support  the  same  analysis. 

There  are  two  lines  of  argument  that  will  serve  as  the  basis  for  discussion.  The  first  argument  is  that  both 
workload  and  situational  awareness  have  been  developed  as  constructs  because  of  the  lack  of  understanding  of 
basic  cognitive  processes  and  if  there  is  an  adequate  model  of  cognition  there  is  no  need  to  consider  these  high 
level  abstractions  in  the  analysis  of  context.  If  this  argument  is  accepted,  the  development  of  good  quality 
cognitive  models  will  eliminate  the  need  to  consider  the  high-level  constructs  and  they  will  gradually  fall  into 
disuse. 

The  second  argument  is  that  these  constructs  provide  a  useful  description  of  human  performance  in  complex 
situations,  providing  a  means  to  test  and  validate  our  models.  If  they  fail  to  describe  what  we  understand  about 
workload,  situational  awareness  and  similar  measures  they  are  clearly  not  useful  models.  At  the  heart  of  this 
discussion  is  what  do  these  output  metrics  measure  and  what  is  their  utility  in  terms  of  modelling  human 
behaviour. 

8.2  DISCUSSION 

8.2.1  Pleasing  the  Customer 

It  was  clear  from  the  discussion  that  some  customers  request  output  from  models  that  is  on  the  level  of 
constructs  such  as  workload  or  SA.  It  was  argued,  therefore,  that  human  modellers  should  provide  these 
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measures,  otherwise  someone  else  (likely  without  an  appropriate  human-sciences  knowledge  base)  will  do  it, 
but  will  probably  do  it  badly.  Another  point  in  favour  of  using  these  constructs  is  that  subjects  do  report,  or  at 
least  are  able  to  report,  experiencing  things  such  as  “workload”.  Our  models  should  therefore  also  generate 
such  subjective  responses. 

It  was  argued  that  we  could  bypass  the  problems  with  these  constructs  if  we  would  not  use  them  as  modelling 
concepts,  but  would  generate  these  fuzzy  constructs  from  the  information  present  in  the  model.  The  main  value 
in  these  concepts  is  as  summary  output  measures  that  communicate  aspects  of  performance  to  stakeholders. 

8.2.2  Terminology 

Most  of  those  participating  in  the  discussion  agreed  that  many  people  outside  of  the  Human  Factors 
community  use  loose  definitions  of  SA,  viewing  it  as  a  concept  solely  related  to  sensors,  data  and  displays 
rather  than  cognitive  processes  or  knowledge  state  associated  with  understanding  of  the  state  of  the  immediate 
environment,  as  defined  by  Endsley  and  others.  HBR  modellers  might  therefore  be  better  off  using  the  term 
“sense  making”  instead,  to  distinguish  SA  from  the  technologies  that  support  the  acquisition  of  SA. 

8.2.3  Containers 

The  general  opinion  was  that  these  constructs  are  actually  containers  representing  more  complex  and 
distributed  phenomena.  Workload  could  probably  better  be  described  as  a  collection  of  bottlenecks,  instead  of 
one  big  bottleneck.  A  similar  argument  holds  for  SA,  since  SA  is  probably  a  distributed  (and  somewhat  fuzzy) 
collection  of  facts  and  interpretations  of  these  facts  that  can  not  be  localized  or  concretized. 

It  was  argued  that  devoting  energy  to  modelling  these  constructs  to  high  fidelity  is  a  poor  investment  of 
resources,  since  they  are  at  best  of  limited  use  as  modelling  constructs.  These  constructs  are  therefore  not  to 
be  used  as  a  starting  point  of  the  modelling  process  but  will  evolve  when  “correctly”  modelling  performance 
(workload)  and  behaviour  (SA). 


8.3  CONCLUSION 

Overall  it  was  agreed  that  workload  and  SA  are  not  useful  as  starting  points  for  the  modelling  process. 
However,  it  was  agreed  that  both  measures  should  be  derived  from  model  outputs  in  a  form  that  can  be  related 
to  rigorously  defined  constructs,  so  that  they  can  be  presented  to  stakeholders  in  a  way  that  communicates 
complex  constructs  clearly.  No  rigorous  definitions  were  agreed  within  the  session,  although  it  was  accepted 
that  workload  is  a  measure  of  the  amount  of  task  activity  that  has  to  be  completed  per  unit  time  and  Situation 
Awareness  encompasses  the  human  quality  of  understanding  as  expressed  in  a  predictive  mental  model  as 
well  as  the  hard  measure  of  what  information  is  available. 
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9.1  INTRODUCTION 

The  concept  of  moderators  is  based  on  the  idea  that  there  is  a  plausible  ‘datum’  level  of  human  behaviour  that 
is  modified  by  variations  in  internal  state  or  the  external  environment.  The  modifications  are  frequently 
viewed  as  the  primary  source  of  human  variability  and,  as  such,  have  to  be  modelled  in  any  nominally 
complete  representation  of  human  behaviour.  The  science  of  the  impact  of  a  range  of  environmental  stressors 
has  been  the  stock-in-trade  of  human  science  research  for  the  past  60  years  and  some  sophisticated  models 
have  been  developed  to  explain  human  response  to  the  stressful  conditions.  Even  fairly  simple  cases  can  lead 
to  considerable  cost  and  complexity  in  implementation. 

A  barrier  to  the  exploitation  of  Human  Behaviour  Representation  in  models  for  the  analysis  of  systems  and  for 
supporting  training  applications  is  the  cost  of  developing  and  validating  the  supporting  elements.  Developing 
a  suite  of  models  of  moderators  is  a  complex  and  expensive  undertaking  that  involves  a  combination  of 
physiological  and  cognitive  models.  Although  many  of  the  models  are  generic  and  can  be  reused,  some 
components  will  be  task  specific  and  have  to  be  tailored  for  each  application,  increasing  the  cost. 

Do  we  have  to  be  scientifically  rigorous  to  develop  models  that  are  fit  for  purpose?  Can  we  identify  the 
number  of  internal  states  that  we  have  to  represent  or  can  we  approach  the  problem  by  using  an  argument  that 
focuses  on  a  single  level  of  “arousal”  to  capture  all  stresses?  Can  we  capture  what  we  need  by  using  a 
stochastic  representation  of  human  behaviour  and  rely  on  sensitivity  analysis  to  draw  our  conclusions? 


9.2  DISCUSSION 

The  term  moderator  is  used  to  express  any  element  in  the  simulation  that  can  take  different  values  on  different 
occasions  and  that,  through  some  route,  affects  human  performance.  To  aid  the  analysis,  moderators  have 
been  classified  as  external,  internal  or  collective,  defined  as: 

•  External:  Elements  in  the  environment  that  can  change  the  psychological  or  physiological  state  of  an 
individual.  Examples  are  temperature,  clothing,  imposed  sleep/rest/work  pattern,  time  of  day,  presence 
of  a  threat,  or  demand  of  an  activity  for  work. 
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•  Internal:  Individual  characteristics  that  can  affect  the  psychological  or  physiological  state  of  an 
individual,  usually  but  not  always  in  conjunction  with  an  external  moderator.  Examples  are  height, 
weight,  percentage  body  fat,  gender,  aspects  of  personality,  experience,  or  level  of  training. 

•  Collective:  Characteristics  of  the  collective  (or  team)  that  can  affect  the  state  of  interactions  between 
the  individuals  who  make  up  the  team  or  collective.  Examples  are  collective  experience,  nature  of 
leadership. 

The  model  of  the  action  of  external  and  internal  moderators  assumes  that  the  moderating  variable  changes  the 
psychological  or  physiological  state  of  an  individual  and  that  this  state  change  is  what  drives  the  change  in 
behaviour  or  performance.  State  is  therefore  seen  as  dynamic  within  a  simulation  in  that  it  changes  with  time. 
In  general  the  internal  moderators  are  viewed  as  “traits”  and  therefore  fixed  with  time  for  a  given  simulation. 
There  is  a  question  of  how  many  independent  psychological  or  physiological  states  can  be  defined  usefully. 
Early  work  on  psychology  identified  a  single  state:  arousal.  For  the  purposes  of  modelling  changes  in 
performance  or  behaviour  a  number  of  states  have  been  defined  that  relate  to  different  moderators.  Some  states 
can  clearly  be  affected  by  more  than  one  moderator.  Some  examples  are  provided  below. 


Moderator 

State(s) 

Environmental  temperature 

Core  temperature 

Clothing 

Skin  temperature 

Height 

Thermal  comfort 

Percentage  body  fat 

Hydration  status 

Exercise  level 

Sweat  rate 

Drinking  water 

Exercise 

Blood  glucose  level 

Food  consumption 

Core  temperature 

Skin  temperature 

Thermal  comfort 

Sweat  rate 

Time  of  day 

Prior  sleep  pattern 

Time  zone  history 

Work/rest  pattern 

Alertness 

External  threat 

Fear/Anxiety 

Personality 

A  range  of  models  has  been  developed  that  describe  the  effect  of  the  environmental  stressors  on  state. 
Examples  are  provided  by  sleep-loss  fatigue  models  and  alertness,  or  thermal  models  and  body  temperature. 
There  is  limited  work  describing  the  relationship  between  internal  state  and  performance  but  some  has  been 
conducted  (Belyavin  &  Spencer,  2004). 
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It  is  clearly  possible  to  define  physiological  states  that  relate  to  external  stressors.  It  is  not  as  simple  to  isolate 
psychological  states  that  relate  to  the  “psychological”  moderators  such  as  personality,  level  of  training, 
experience.  Arousal  has  been  used  by  psychologists  for  many  years  and  appears  to  be  a  useful  way  of 
interpreting  a  number  of  effects.  It  is  not  clear  whether  fear  or  anxiety  represents  a  distinct  state  or  whether  it 
drives  arousal  and  affects  performance  by  this  route.  A  research  requirement  is  the  definition  of  the  number  of 
different  states  that  need  to  be  considered  for  the  modelling  of  performance  and  behaviour  and  how  these 
relate  to  the  moderators  of  interest. 

There  is  considerable  discussion  as  to  whether  the  problem  of  modelling  more  than  one  set  of  moderators 
simultaneously  is  more  complex  than  modelling  single  moderators  due  to  interactions  between  the  associated 
states.  An  example  would  be  the  combination  of  the  moderators  that  make  up  the  thermal  environment  and 
those  relating  to  sleep  loss.  If  the  sets  of  moderators  do  not  interact,  the  effect  of  a  change  in  alertness  on 
performance  or  behaviour  is  not  affected  by  current  thermal  status.  Interaction  implies  that  the  different  states 
affect  the  differing  relationships  between  performance  and  behaviour.  Preliminary  investigation  of  a  number 
of  external  moderators  indicates  that  interactions  of  this  kind  may  be  present  but  are  not  large. 

Andy  Belyavin  indicated  that  analysis  at  QinetiQ  identified  relatively  few  interactions  between  moderators, 
based  on  a  review  of  the  relevant  literature.  It  was  noted  that  interactions  between  internal  and  external 
moderators  (flight  strategy  and  fatigue)  have  been  observed  in  basic  research  using  computational  cognitive 
models  to  study  aircraft  manoeuvring.  Expert  flight  strategies,  such  as  the  control  and  performance  concept 
for  instrument  flight,  appear  more  fatigue  resistant  than  novice  strategies. 

The  challenge  is  to  choose  a  minimally  sufficient  set  of  state  variables  and  to  model  their  impact  on 
performance.  There  are  a  number  of  key  questions. 

•  Could  we  use  similar  approaches  to  represent  analytical  physiological,  cognitive  and  emotional 
processes? 

•  Are  there  valid  models  of  individual  states? 

•  What  would  be  the  potential  interactions  among  the  states? 

Laurel  Allender  described  how  moderators  are  represented  in  IMPRINT  (for  more  information,  see:  Allender, 
Salvi  &  Promisel,  1997).  An  engineering  model  is  used  based  on  performance  data  that  defines  a  direct 
relationship  between  moderator  and  performance.  The  task  is  broken  out  over  a  taxonomy  of  lower  level 
components  such  as  gross  motor  movement,  fine  motor  movement,  visual  perception,  the  components 
degraded  and  the  outcome  reconstructed  from  the  components.  The  basis  of  the  taxonomy  is  the  mental  or 
physical  resource  employed.  For  example  in  a  decision  making  task  the  main  resource  would  be  the  mental 
resource.  It  was  observed  that  extrapolation  from  a  data  based  model  of  this  kind  outside  the  range  of  the 
source  data  is  not  reliable  and  that  more  validity  may  be  expected  from  a  model  that  includes  the  intervening 
state  and  trait  variables  based  on  sound  scientific  relationships  rather  than  mapping  the  moderator  directly 
onto  performance. 

It  was  noted  that  the  representation  of  moderators  is  linked  to  architecture  because  they  form  fundamental 
architectural  constraints.  Representing  the  effects  of  moderators  in  some  architectures  is  difficult.  The  design 
of  architectures  should  enable  valid  models  of  the  effects  of  moderators  to  be  incorporated. 

There  was  a  discussion  of  what  was  needed  from  modelling  the  effects  of  moderators.  For  training  applications  a 
view  was  expressed  that  plausible  variation  in  behaviour  was  sufficient  to  meet  the  training  need.  The  majority 
view  of  the  meeting  was  that  this  is  an  ill-advised  approach  as  it  would  result  in  TLAR  (That  Looks  About 
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Right)  validity  that  could  easily  fail  for  conditions  only  marginally  different  from  the  original  design 
assumptions.  Crude  validity  of  this  kind  may  be  appropriate  for  narrowly  focused  applications,  but  for  serious 
system  analysis,  general  training  or  mission  rehearsal  where  a  broad  spectrum  of  conditions  could  be  anticipated, 
scientifically  sound  models  and  a  higher  level  of  validity  are  required. 


9.3  CONCLUSION 

Overall  the  meeting  supported  the  argument  that  validated  models  of  the  effects  of  moderators  are  an  essential 
component  of  all  HBR  applications.  The  approach  involving  external/intemal/collective  moderators  related  to 
internal  states  was  endorsed  in  principle,  although  there  was  no  agreement  on  an  exhaustive  list  of  either 
moderators  or  states.  It  was  agreed  that  model  architectures  need  to  take  account  of  modelling  the  effects  of 
moderators  explicitly. 
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10.1  INTRODUCTION 

In  Chapter  8  of  The  Sciences  of  the  Artificial ,  Herb  Simon  (1969)  described  the  role  of  hierarchy  in  complex 
systems.  Four  key  points  include: 

1)  Complex  systems  are  nearly  always  hierarchic; 

2)  Hierarchy  facilitates  evolution; 

3)  Near  decomposability  is  an  important  characteristic  of  the  hierarchical  components  of  complex  systems; 
and 

4)  Nearly  decomposable,  hierarchic  structures  facilitate  comprehension. 

In  this  session  we  considered  HBR  architectures  from  the  perspective  of  nearly  decomposable,  hierarchical, 
evolutionary,  complex  systems. 


10.2  SPEAKER  PRESENTATIONS 

Randy  Jones,  SOAR  Technology  Ltd. 

Architectures  and  frameworks  have  higher  level  hypotheses  and  theories  that  govern  them  but  there  is  a  lot  of 
devil  in  the  detail.  Modelling  is  often  much  more  complex  than  frameworks  suggest.  To  complicate  matters, 
different  architectures  have  different  names  for  quite  similar  concepts,  processes,  and  mechanisms.  None  of 
the  architectures  have  all  the  mid-level  constructs  but  they  all  have  several  similar  key  elements.  Modelled 
tasks  all  have  a  somewhat  similar  knowledge  structure  across  architectures.  A  shared  meaning  of  terms  and  a 
common  vocabulary  would  be  useful  in  the  community. 

No  architecture  supports  all  basic  representations  and  processes  needed  for  all  HBR  needs.  Randy  expressed  the 
need  for  a  comparative  framework  to  compare  functions  and  features  of  all  different  architectures.  To  highlight 
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the  challenge,  he  noted  that  in  Pew  and  Mavor  (1998),  major  cognitive  architectures  are  described  in  the  words 
of  the  developers  who  use  different  terminology  for  fundamentally  similar  architectural  components.  Therefore, 
a  deep  understanding  of  each  architecture  is  required  to  know  which  terms  and  functions  are  synonyms  and 
which  are  not. 

Randy  described  his  interests  and  experiences  in  identifying  similarities  in  processes  and  representations 
across  cognitive  architectures.  His  initial  analysis  included  BDI,  GOMS,  and  Soar.  Randy  published  this 
initial  work  in  AI  Magazine  in  2006.  More  recently  he  and  colleagues  at  Soar,  in  collaboration  with  Christian 
Lebiere  at  Carnegie  Mellon,  developed  an  abstraction  language  called  High-Level  Symbolic  Representation 
(HLSR)  and  have  been  investigating  the  use  of  HLSR  to  simultaneously  generate  Soar  and  ACT-R  models  of 
the  same  task.  They  see  HLSR  as  part  of  their  long-term  strategy  to  create  a  modular,  hierarchical 
implementation  language  that  is  useful  for  multiple  architectures. 


Dr.  Emiel  Ubink,  TNO 

Modular  HBR :  The  many  parts  of  a  human  model  will  in  most  cases  be  constructed  as  a  nearly 
decomposable  hierarchy,  as  described  by  Herb  Simon  (see  introduction).  It  seems,  however,  that  most 
behavioural  architectures  are  an  exception  to  this  rule,  at  least  at  the  functional  or  behavioural  level.  In  most 
behavioural  architectures  several  components  (blackboard,  rule  based,  etc.)  have  to  be  consulted  to  generate  a 
specific  type  of  behaviour,  such  as  answering  the  telephone.  This  shows  that  on  a  functional  level  the 
behavioural  architecture  is  not  at  all  nearly  decomposable.  In  Emiel’ s  behavioural  architecture  based  on 
pandemonium  theory  the  components  (demons  in  the  pandemonium)  are  functional  units  (“behaviour 
chunks”)  instead  of,  for  instance,  units  corresponding  to  the  components  in  a  theory  of  cognition.  The  demons 
in  this  pandemonium  are  in  competition  over  a  limited  set  of  resources  that  represent  the  capabilities  of  the 
system  and  that  are  needed  to  generate  behaviour.  This  results  in  a  nearly  decomposable  system  that  simplifies 
the  development  of  complex  behaviour,  that  improves  the  maintainability,  and  that  allows  one  to  easily  plug 
in  or  remove  specific  behaviours. 

Variability  in  Behaviour :  Variability  in  behaviour  has  to  do  with  action  selection  and  action  execution 
(i.e.,  which  actions  are  taken  and  how  these  actions  are  executed).  These  “what  to  do”  and  “how  to  do  it” 
questions  are  often  addressed  separately.  They  are,  however,  very  much  intertwined,  especially  with  respect  to 
human  performance  modelling.  The  answer  to  the  “what”  question  could  very  well  be  influenced  by  the  “how” 
answer  (since  I  can  not  run  fast  enough  I  will  not  run  at  all)  and  vice  versa  (if  I  decide  to  move  to  location  A, 
I  will  have  to  run).  These  “what”  and  “how”  questions  are  so  entangled  because  of  the  many  feedback  loops  in 
behaviour,  and  because  both  the  “what”  and  the  “how”  of  behaviour  depend  for  a  large  part  on  the  capabilities  of 
the  modelled  system,  that  should  be  viewed  as  a  limited  set  of  resources.  The  conclusion  is  therefore  that  when 
modelling  human  behaviour  a  way  needs  to  be  found  to  address  both  questions  simultaneously. 

A  portion  of  Emiel’ s  presentation  focused  on  the  relation  of  decomposability  to  validity.  A  fully  decomposable 
architecture  can  be  validated  at  the  element  level.  This  served  as  a  nice  segue  into  Robert  West’s  presentation. 

Dr.  Robert  West,  Carleton  University 

Robert  emphasized  the  point  that  evaluations  of  the  validity  of  models  and  architectures  are  required  in  order 
to  achieve  objective,  measurable  progress  in  HBR.  He  and  his  colleagues  at  Carleton  University  are  re¬ 
implementing  ACT-R  in  Python  using  modular  components  that  can  easily  be  modified  to  change  the  system 
architecture.  This  facilitates  modification  and  validation  of  the  separate  modules  comprising  the  architecture. 
Python  was  chosen  to  ease  code  writing  instead  of  using  LISP,  ACT-R’ s  native  language.  To  make  progress, 
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we  need  to  show  validity  in  the  approach  and  the  models  developed,  including  predictive  validity, 
but  validation  is  an  expensive  process  that  the  civilian  market  seems  unwilling  to  assume.  Without  validation, 
we  are  left  with  curve-fitting  and  the  best  curve-fit  solution  isn’t  necessarily  the  one  that  captures  the  essence 
of  the  data  best.  Architectures,  or  rather  models  built  in  architectures,  have  to  accurately  predict  human 
behaviour. 

Robert  expressed  an  interest  in  testing  model  fit  against  different  datasets  and  established  parameter  values. 
Better  ways  of  fitting  model  output  to  empirical  data  are  also  needed. 


10.3  DISCUSSION 

The  general  discussion  focused  primarily  on  modularity,  common  resources,  and  validity.  Here  are  some 
highlights  from  the  points  and  questions  raised. 

10.3.1  Modularity 

Object  oriented  programming,  service  based  software  architectures,  modular  thinking  (Randy  mentioned 
aspect  oriented  programming)  is  the  trend  in  all  software  development.  The  need  for  development  of  middle 
level  constructs  and  functionality  was  raised  by  both  Randy  and  Emiel. 

10.3.2  Common  Resources 

Architectures  for  HBR  need  to  be  free  and  open  source  to  promote  reuse,  particularly  by  academia,  which  is 
well  positioned  to  address  the  validation  question.  There  is  a  need  to  develop  open,  common  agents  that  do 
things  so  that  the  community  can  explore  them  critically. 

An  on-line  resource  model-repository  would  be  useful.  The  NATO  RTO  SAS  053  Virtual  Institute  for  HBR 
came  up  in  the  discussion,  but  there  was  some  confusion  as  to  its  status.  Brian  Gore  pointed  to  the  need  for  a 
place  to  publish  new  models.  He  mentioned  a  new  journal  coming  out:  the  International  Journal  of  Human 
Factors  in  Modelling  and  Simulation  (Brian  Gore  is  an  editor),  providing  details  much  as  the  AMBR  project 
did;  NASA  also  has  a  book  out  covering  model  validation  similar  to  AMBR  but  in  a  different  context, 
possibly  with  a  web-based  repository  also  connected  to  it.  Also  the  journal  Cognitive  Engineering  and 
Decision  Support  was  mentioned  as  a  place  to  publish  models. 

10.3.3  Validity 

It  can  be  difficult  to  differentiate  between  a  good  architecture  and  a  good  model.  How  do  you  validate  an 
architecture  for  predictive  validity?  Perhaps  construct  validity  and  content  validity  are  sufficient  for 
architectures  unless  some  unique  aspect  of  the  architecture  forces  the  model  to  predict  a  certain  observation. 
Lots  of  successful  models  built  in  an  architecture  lends  credence  to  its  validity  as  an  architecture,  but  you  have 
to  ensure  that  the  models  follow  the  tenets  of  the  architecture  and  do  not  simply  develop  convenient  shortcuts 
or  work-arounds  to  simplify  the  model  development. 

Comparative  validity  is  difficult  in  the  context  of  architectures  and  HBR-based  models  because  only  the 
developers  really  understand  the  model  implementations  and  approximations,  and  sometimes  even  the 
individual  developers  don’t  have  a  complete  understanding  of  the  entire  architecture  in  sufficient  depth.  It  is 
hard  to  compare  models  of  the  same  phenomena  if  design  decisions  are  not  documented. 
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Can  we  validate  architectures  or  can  only  specific  models  be  validated?  An  architecture  can  only  be  validated 
through  its  use  in  many  studies  (i.e.,  validated  on  a  meta-level).  Evidence  accrues  for  the  validity  of  an 
architecture  just  as  evidence  accrues  for  the  validity  of  scientific  theories. 

Validity,  whether  at  the  model  level  or  the  architecture/system  level  is  not  a  binary  state.  Validation  is  a 
process  of  accrual  of  evidence  and  validity  is  a  continuum  rather  than  binary  state  that  is  context  dependent. 

10.4  SESSION  RECOMMENDATIONS  FOR  HBR  S&T 

We  need  datasets  from  a  wide  and  increasing  range  of  tasks  in  order  to  objectively  establish  the  broader 
validity  of  the  cognitive  architectures  and  HBR  systems  under  development  today. 

Some  type  of  middleware  that  facilitates  cross-HBR  comparison  is  advisable. 

The  HBR  R&D  community  would  likely  benefit  from  a  better  understanding  of  productive  methods  for 
comparing  models  across  architectures  and  comparing  architectures. 

Continuing  the  trend  toward  modular  HBR  systems  and  cognitive  architectures  is  advisable. 


10.5  SESSION  RECOMMENDATIONS  FOR  NATO 

There  should  be  some  interaction  between  SAS  and  HFM  to  explore  potential  for  collaboration  on  HBR. 
The  NATO  RTO  SAS-053  Virtual  Institute  for  HBR,  which  was  previously  recommended  by  RTO  SAS-017 
panel,  is  due  to  wrap  up  in  2007,  but  information  is  required  such  as  the  timeline  for  standing  up  the  Virtual 
Institute,  its  envisioned  capabilities  and  suspected  limitations.  The  Virtual  Institute  for  HBR  may  provide  a 
natural  site  for  locating  a  repository  of  models  and  data  that  can  be  made  public  for  modellers  in  the  field. 
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11.1  INTRODUCTION 

Human  factors  are  often  considered  at  the  individual  level  but  military  operations  are  typically  a  team  effort. 
Early  analytical  simulation  of  large-team,  force-on-force  engagements  (Lanchester,  1916)  contained  little  that 
could  be  considered  Human  Factors.  This  trend  seems  to  have  persisted  into  more  modern  simulations  despite 
an  interest  in  small  team  activities  within  an  operation.  Even  at  the  high  aggregation  levels,  human  aspects  of 
collections  of  individuals  such  as  morale  and  commitment  are  thought  to  play  a  significant  role  in  areas  of 
interest  to  Operations  Research. 


11.2  DISCUSSION 

At  the  small  team  level  (fire  team,  platoon,  company,  flight  crew,  squadron,  etc.),  individual  Human  Factors  are 
still  relevant  (for  example,  workload,  fatigue,  thermal  strain,  etc.),  however,  other  properties,  characteristics  and 
behaviours  appear  that  experts  feel  are  important  to  operation  success  (for  example,  morale,  mutual  support, 
commitment,  etc.).  While  these  are  still  attributes  of  individuals,  they  arise  because  of  the  interaction  of  the 
individuals  in  the  team  in  their  coordinated  pursuit  of  a  common  goal.  Other  behavioural  markers  are  also 
present  in  small  teams  that  may  be  overlooked  at  higher  levels  of  aggregation,  but  are  important  to  capture  for 
team  simulation.  These  behaviours  include  communication  and  feedback,  mutual  support  (backing  up)  and  task 
sharing,  team  assessment  and  monitoring  of  teammates,  to  name  a  few. 

Teams  represent  an  additional  level  of  complexity  over  individuals  so  it  is  reasonable  to  be  concerned  about 
appropriate  representation  when  we  have  trouble  modelling  individuals,  but  if  the  degrees  of  freedom  of  the 
analysis  can  be  controlled,  and  the  subsystems  suitably  characterized,  then  team  modelling  seems  feasible. 
Brad  Cain  suggested  that  the  concern  about  the  additional  complexity  doesn’t  necessarily  follow  as, 
by  analogy,  many  aspects  of  fluid  dynamics  may  be  captured  at  a  continuum  level  without  modelling  the 
dynamics  of  individual  molecules.  This  reflects  capturing  individual  HBR  effects,  such  as  workload,  as  team 
latent  variables  that  can  be  considered  factors  at  an  aggregate  level,  much  the  same  as  viscosity  reflects  the 
exchange  of  momentum  of  molecules  for  fluids  at  a  continuum  level.  Andy  Belyavin  remarked  that  individual 
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representation  may  be  necessary  to  detect  certain  phenomena  but  there  should  be  nothing  to  restrict  a  mixture 
of  aggregate  and  individual  entities  as  long  as  the  interaction  phenomena  are  represented  appropriately. 

Team  models  can  be  conceived  as  a  collection  of  cooperating  individual  models  or  as  a  team-entity  that  entails 
no  distinct  representation  of  its  constituent  members.  Each  representation  approach  has  implications  for  the 
level  of  representation  used  and  the  human  sciences  requirements.  Melanie  Linde  reported  that  some 
combination  of  these  two  approaches  has  been  successfully  used  in  Germany,  where  communication  and 
group  reasoning  within  teams  has  been  represented  as  a  “disembodied  leader”  construct.  The  “disembodied 
leader”  -  assumed  to  be  a  non-human,  mechanistic  process  model  -  may  preclude  the  interaction  of  operator 
states  such  as  workload  effects  on  group  reasoning  and  leader  activities.  If  timing  is  critical  in  the  analysis,  the 
disembodied  leader  approach  also  may  cause  problems  representing  where  group  knowledge  lies  within  its 
member.  It  was  also  observed  that  “blobs”  with  disembodied  leaders  would  not  be  appropriate  for  modelling 
the  impact  of  complex  interfaces.  Nevertheless,  the  disembodied  leader  approach  may  be  a  suitable 
approximation  when  the  desired  level  of  granularity  of  the  analysis  is  somewhere  between  a  collection  of 
individuals  and  the  team  entity  level. 

The  collection  of  individuals  approach  allows  more  detailed  models  of  the  individual  behaviours,  capabilities 
and  limitations.  Conceivably,  the  collection  of  individuals  approach  can  provide  insight  into  why  team 
performance  is  good  or  breaks  down  but  this  in  turn  requires  more  effort  to  model  the  interpersonal 
interactions;  that  is,  the  teamwork.  But  Rob  West  reported  their  observation  that  teamwork  is  often 
characterized  by  frequent  interruptions  and  task  switching,  features  that  are  not  well  captured  by  standard 
GOMS  or  task  analytic  approaches  that  focus  on  individual  jobs.  These  analytical  techniques,  while  capturing 
the  task-work  often  fail  to  capture  the  team-work  that  leads  to  role  and  resource  switching.  Such  phenomena 
might  be  better  captured  with  a  mixture  of  HTA  and  TNO’s  pandemonium  model,  or  some  other  task 
prioritization  scheme,  to  reflect  changing  attentional  foci  of  team  members. 

Laurel  Allender  suggested  that  dynamic  network  analysis  might  be  a  useful  approach  to  capture  interactions 
between  team  members  as  it  measures  both  events  and  the  content  of  any  communications  and  enables 
analysis  of  communications  in  detail.  The  collection  of  individual  models  need  to  include  the  team  tasks  and 
include  task  scheduling  that  that  dynamic  network  analyses  captures  to  represent  the  ability  or  tendency  of 
individuals  to  deviate  from  a  plan  in  a  constraint  based  system  by  opportunistic  replanning. 

The  team  entity  modelling  approach  focuses  above  the  level  of  the  individual  behaviours  or  abilities,  and  is 
the  mainstay  of  many  Operations  Research  military  analyses.  Rob  West  observed  that  details  of  the  individual 
actions  within  the  team  by  its  members  are  typically  ignored  in  organisational  dynamics  modelling. 
Organizational  modelling  is  focused  at  a  high-level  of  abstraction,  examining  the  effects  of  performance 
variability,  not  why  the  performance  actually  degrades.  The  team  entity  approach  loses  some  of  the  resource 
management  constraints  (such  as  cognitive  workload)  simplifying  the  modelling,  but  it  introduces  new  latent 
variables  that  are  team  characteristics.  These  latent  variables  are  often  not  formally  defined  -  Are  there 
representative  models  of  how  these  latent  teamwork  aspects  such  as  communication  and  situation  awareness, 
perception  and  workload  evolve  with  events? 

There  seemed  to  be  a  consensus  that  we  can  define  the  essential  aspects  of  team  performance  that  can  be 
modelled  and  measured.  Carol  Cooper-Chapman  reported  that  the  UK  has  developed  conceptual  models  of 
how  some  of  these  factors  interact  and  the  dstl  work  on  STORM  has  explored  social  and  cultural  effects 
through  formal  models  in  the  analysis  of  collective  behaviour.  There  appears  to  be  a  need  to  delve  further  into 
the  team  research  field  to  see  if  there  are  sufficient  data  that  formal  models  can  be  further  developed, 
establishing  the  distinction  between  Teamwork  (interactions  between  team  members  that  are  an  overhead)  and 
Taskwork  (individual  work  typically  captured  by  GOMS-like  analyses). 
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Team  modelling  might  benefit  from  experimentation  in  games  (such  as  DARPA’s  “Realworld1”  and 
Breakaway’s  “A  force  more  powerful2”.  Bob  Foster  noted  that  NATO  has  collective  training  centres  that  might 
be  a  good  source  of  observational  data  to  capture  the  interaction  of  different  cultures  in  a  team  of  individuals 
(command  centre)  or  the  cooperative  activities  of  brigades  from  different  nations. 

It  was  thought  that  many  of  the  aspects  of  crowds  can  be  modelled  at  the  fluid  dynamics  level.  There  has  been 
some  success  in  inserting  beliefs  and  states  into  crowds  to  modify  their  behaviour.  It  was  noted  that  crowds 
might  be  considered  an  extreme  form  of  organisations  and  Old  Dominion  University  (among  others)  has  looked 
at  the  roles  of  individuals  in  the  crowd.  A  military  organisation  might  be  represented  as  a  hierarchy  of  small 
structured  teams  but  the  modelling  problem  area  lies  between  these  two  extremes  -  “edge”  organisations  - 
where  individuals  in  the  crowd  play  roles  as  do  small  groups  that  might  have  a  team  attributes  yet  their 
interaction  in  the  crowd  entity  is  uncoordinated.  Jerome  Levesque  described  some  preliminary  work  to  develop  a 
synthetic  urban  environment  with  crowd  modelling  to  study  and  train  for  emergency  response  (discussion  in 
section  4  below)  that  would  benefit  from  data  derived  from  observing  domestic  crowds  (such  as  Old  Dominion’s 
research)  as  well  as  from  NATO  training  centres  if  they  engage  in  Effects  Based  Operations  training. 


11.3  CONCLUSION 

There  seemed  to  be  a  consensus  that  team  modelling  is  viable  with  current  technology.  The  HBR  modelling 
community  could,  with  some  additional  study,  define  the  pros  and  cons  of  the  various  approaches  to  team 
modelling:  capturing  teams  as  blobs,  collections  of  cooperating  individuals  or  some  combination  of  the  two. 
Such  a  study  would  serve  the  M&S  community,  making  recommendations  about  the  suitability  of  each 
method  for  different  purposes.  Unfortunately,  there  doesn’t  seem  to  be  a  ready  source  of  these  pros  and  cons 
so  some  additional  study  is  required  to  elicit  this  information  from  users  of  these  techniques  or  by  a  team 
modelling  panel.  There  was  no  indication  of  whether  these  approaches  are  affordable,  but  since  many 
organizations  are  engaged  in  some  form  of  team  analyses,  it  would  appear  that  some  levels  of  representation 
are  affordable.  Validation  of  team  models  remains  a  tricky  issue,  although,  as  previously  suggested  by 
Lochlan  Magee,  transfer  of  training  may  provide  a  methodology  for  validation  at  some  practical  level. 


11.4  SESSION  RECOMMENDATIONS 

1)  Develop  guidance  to  the  M&S  community  on  the  appropriateness  of  various  team  and  crowd  modelling 
approaches  to  application  areas. 

2)  Explore  how  task-analytical  techniques  can  be  extended  to  include  teammork  to  support  modelling  teams 
as  collections  of  individuals. 

3)  Document  formal  models  that  attempt  to  capture  the  effects  of  team  and  crowd  latent  variables  on 
behaviour. 

4)  Explore  using  NATO  training  facilities  as  a  source  of  data  to  support  team  modelling. 


1  http://www.darpa.mil/ipto/programs/real/index.htm 

2 

http :// w  w  w .  afmpg  ame  .com/ 
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11.5  PRESENTATION  SUMMARIES 

CAMICS:  Civilian  Activity  Modelling  in  Military  Constructive  Simulation 

Jerome  Levesque*,  Allister  MacIntyre**,  Greg  Phillips***,  Robert  St-John** 

*  DRDC-CORA,  DGLCD/Building  A-31,  PO  Box  17000,  Station  Forces,  Kingston,  Ontario  K7K  7B4,  Canada 

**  Dept,  of  Psychology.,  Royal  Military  College  of  Canada,  Kingston,  Ontario  K7K  7B4,  Canada 

***  Dept,  of  Electrical  and  Comp.  Eng.,  Royal  Military  College  of  Canada,  Kingston,  Ontario  K7K  7B4,  Canada 

This  is  a  summary  for  a  project  that  has  started  recently  (June  2006).  It  is  funded  by  the  Centre  for  Operational 
Research  and  Analysis  (CORA),  a  research  organization  part  of  Defence  Research  and  Development  Canada 
(DRDC).  Our  goal  is  to  use  the  current  knowledge  in  human  behaviour  representation  and  traffic  modelling  to 
implement  a  model  of  civilian  population  activity.  The  model  will  be  used  in  constructive  simulations  for 
training  and  experimentation. 

11.5.1  Aims 

To  develop  a  software  model  of  civilian  activity  for  military  synthetic  training  environments,  with  a  focus  on 
urban  areas. 

To  implement  behaviours  that  are  coherent  with  current  literature  and  research  in  psychology  and  routing 
algorithms  that  reproduce  basic  urban  traffic  features. 

11.5.2  Background 

Simulations  such  as  OneSAF,  JCATS,  and  JSAF  are  currently  used  to  represent  a  wide  variety  of  traditional 
operational  and  war  fighting  scenarios.  Recently,  simulations  have  begun  to  be  used  to  train  for  counter¬ 
insurgency,  urban  peacekeeping,  crowd  control  situations,  and  other  non-traditional  operations.  As  a 
consequence,  new  models  have  emerged  that  attempt  to  simulate  crowd  and  group  behaviours  (e.g.,  Crowd 
Federate™).  Scenarios  set  in  urban  environments  do  not  always  involve  crowd  confrontation  though.  To  our 
knowledge,  none  of  the  current  models  deal  with  normal  daily  activity,  such  as  city  traffic  and  busy  street 
corners,  or  with  crowd  formation  itself. 

Existing  crowd  models  often  attempt  to  use  “crowd  psychology”  as  the  basis  for  controlling  simulated  crowd 
behaviours.  However,  crowd  psychology  is  mainly  exhibited  through  varying  levels  of  crowd  aggression. 
The  models  track  individuals  within  the  crowd,  but  limit  the  behaviours  depending  upon  initial  group 
assignment  (e.g.,  an  agitator  group).  That  approach  cannot  be  applied  to  a  model  that  would  include  normal 
daily  behaviour,  where  each  person  remains  driven  by  individual  goals.  Modem  psychological  research 
strongly  suggests  that  each  person  maintains  individual  cognitions,  personalities,  and  beliefs,  (including 
religious  and  cultural  ones),  while  engaged  in  group  behaviours.  Each  person  evaluates  the  nature  of  the 
sub-groups,  the  leadership,  group  structure  and  size,  and  continually  examines  possible  situational  factors  that 
may  influence  their  individual  actions.  This  suggests  that  normal  individual  behaviours  and  group  behaviours 
could  be  included  in  a  single  model. 

In  an  urban  context,  human  circulation  by  foot  is  coexisting  with  normal  vehicle  traffic.  Fortunately, 
sophisticated  models  already  exist  to  model  vehicle  routing,  and  the  implementation  of  only  a  few  basic  rules 
should  be  enough  for  the  training  model  we  propose.  On  the  software  side,  packages  developed  for  the  game 
industry,  such  as  ALimplant™,  now  allow  agent-based  models  of  civilian  activity  to  be  built  easily,  including 
behaviour  models  and  vehicle  routing. 
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The  project  aims  at  providing  the  CF  with  proper  models  of  civilian  activity  for  constructive  simulation 
environments,  with  a  focus  both  on  training  and  experimentation.  It  is  a  collaborative  effort  among  the 
disciplines  of  psychology,  computer  science  and  military  operational  research. 

11.5.3  Project  Description 

This  project  consists  of  two  main  branches: 

1)  Survey  of  current  research  covering  individual  versus  collective  behaviours,  as  well  as  urban  traffic;  and 

2)  Implementation  of  the  models  in  a  software  simulation  to  use  in  constructive  simulations. 

The  progress  of  these  branches  will  overlap  in  time.  It  is  planned  that  a  working  software  prototype  will  be 
developed  rapidly  within  the  first  few  months  of  the  study,  and  be  improved  subsequently  as  research 
progresses.  The  full  research  program  would  eventually  span  a  3-year  period,  each  year  adding  an  increment 
of  functionality  and  reliability  to  the  model,  including  an  extensive  validation  process.  In  the  first  year  it  will 
be  necessary  to  build  the  initial  version  of  the  software  and  implement  basic  behaviour  and  traffic  models. 
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Chapter  12  -  TECHNICAL  EVALUATION  REPORT 

Mr.  William  Fraser 

Human  Factors  Engineering  Group 
Human  Systems  Integration  Section 
Defence  Research  and  Development  Canada  -  Toronto 
P.O.  Box  2000,  1133  Sheppard  Ave.  West 
Toronto,  Ontario  M3M  3B9 
CANADA 


12.1  EXECUTIVE  SUMMARY 

The  Specialists’  Meeting  HFM-143/RSM  on  Human  Behaviour  Representation  (HBR)  in  Constructive 
Modelling  was  held  at  the  Canadian  Forces  College,  Toronto,  Canada  from  30-31  May,  2007.  The  goal  of  the 
meeting  was  to  obtain  input  from  the  scientific  and  operational  communities  regarding  the  validity  of  the 
recommendations  made  by  the  HFM-128  Task  Group  and  to  solicit  input  in  those  areas  where  the  HFM-128 
Task  Group  lacked  experience.  The  meeting  was  organized  around  seven  topic  areas  of  importance  in 
developing  useful  human  behaviours: 

1)  What  human  factors  does  the  operation  involve; 

2)  Human  task  representation  in  M&S; 

3)  Behaviour  generation  -  variability  and  choice; 

4)  Military  relevant  mental  output  measures; 

5)  The  concept  of  moderators; 

6)  Complexity,  hierarchy,  modularity,  and  validity  in  HBR  Architectures;  and 

7)  From  individual  to  group  behaviour. 

The  structure  of  the  meeting,  with  the  focus  on  modelling  issues  rather  than  presentations  on  specific  modeling 
tools  or  models,  provided  the  forum  and  the  time  for  a  wide-ranging  discussion  of  the  many  and  complex 
problems  and  challenges  faced  by  those  involved  in  Human  Behaviour  Representation  in  constructive  modelling 
and  simulation.  One  of  the  central  themes  arising  from  the  presentations  and  the  discussions  is  the  increasing 
need  to  incorporate  a  wide-range  of  Human  Behavioural  Representations  into  constructive  simulations  of 
complex,  non-kinetic,  operations  involving  a  mix  of  military  and  civilian  organizations.  It  is  also  apparent  that 
there  is  a  serious  deficiency  in  tools  for  advanced  model  development,  model  sharing,  simulation,  and  validation. 
To  address  these  and  other  issues,  NATO  RTO  should  initiate  a  series  of  initiatives  to: 

1)  Address  the  development  of  advanced  modelling  tools  to  expand  the  capabilities  in  simulating 
individuals,  individuals  within  teams  and  organizations,  and  teams  and  organizations; 

2)  Develop  new  approaches  to  performance  and  behaviour  metrics  in  large  scale  simulations  and  real- 
world  operations; 

3)  Develop  mechanisms  to  share  data  within  the  NATO  community  that  has  been  collected  from 
both  small  and  large  scale  real-world  operations,  human-in-the-loop  simulations,  and  constructive 
simulations,  to  aid  in  model  development  and  validation;  and 
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4)  Develop  technologies  for  large  scale  automated  data  reduction,  analysis,  and  display,  utilizing 
advances  in  GRID  technologies. 

12.2  INTRODUCTION 

The  Specialists’  Meeting  HFM-143/RSM  on  Human  Behaviour  Representation  (HBR)  in  Constructive 
Modelling  was  held  at  the  Canadian  Forces  College,  Toronto,  Canada  from  30-31  May,  2007.  The  panel  was 
organized  by  the  Human  Factors  and  Medicine  Panel  in  coordination  with  the  members  of  the  HFM-128/ 
RTG. 

Constructive  simulation  is  becoming  an  ever  more  popular  and  powerful  tool  for  use  in  Modelling  and 
Simulation  environments  within  NATO  community.  The  representation  of  human  behaviour  and  performance 
in  these  constructive  simulations  is  a  critical  factor  in  the  use  of  the  technology  to  enhance  the  capability  of 
the  overall  M&S  environment.  However,  a  large  number  of  difficult  and  challenging  issues  surround  the 
development  and  implementation  of  these  representations.  The  purpose  of  the  HFM-128  Task  Group  is  to 
make  recommendations  on  the  implications  of  human  factors  and  cognitive  science  to  improve  the 
implementation  of  human  behaviour  representations  in  complex  simulations.  The  goal  of  the  HFM-143 
Specialists’  Meeting  was  to  obtain  input  from  the  scientific  and  operational  communities  regarding  the 
validity  of  the  recommendations  made  by  the  HFM-128  Task  Group  and  to  solicit  input  in  those  areas  where 
the  HFM-128  Task  Group  lacked  experience. 

In  order  to  simulate  discussions  and  to  provide  focused  feedback  to  the  HFM-128  Task  Group,  the  meeting 
was  organized  around  seven  topic  areas  of  importance: 

1)  What  human  factors  does  the  operation  involve; 

2)  Human  task  representation  in  M&S; 

3)  Behaviour  generation  -  variability  and  choice; 

4)  Military  relevant  mental  output  measures; 

5)  The  concept  of  moderators; 

6)  Complexity,  hierarchy,  modularity,  and  validity  in  HBR  Architectures;  and 

7)  From  individual  to  group  behaviour. 

Formal  presentations  were  relatively  few  and  short.  Instead,  introductory  comments  by  the  Chair  of  each 
session  were  designed  to  stimulate  discussion  and  feedback  on  the  topics  of  concern.  This  technical  review 
has  attempted  to  focus  on  some  general  themes  that  emerged  during  the  two  days  of  discussion  which  suggest 
future  areas  of  research  focus  for  NATO  RTO.  Several  specific  topics  for  follow  on  Tasking  Groups  are 
identified. 


12.3  SUMMARY 

The  first  day  of  the  meeting  started  with  an  introduction  by  Dr.  Lotens,  Chair  of  the  HFM-128  Task  Group 
and  a  Keynote  address  by  Dr.  Foster,  Director,  BioSystems,  Office  of  the  Director,  Defense  Research  and 
Engineering,  US  Dept  of  Defense.  The  second  day  of  the  meeting  was  introduced  with  a  second  Keynote 
address,  by  Mike  Greenley,  Vice  President  of  Modelling  and  Simulation,  CAE. 
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Dr.  Lotens’  opening  remarks  provided  a  clear  focus  for  the  workshop,  outlining  the  goals  and  the  desired 
output  to  assist  the  panel  in  making  recommendations  to  NATO  concerning  simulation  of  human  behaviour  in 
a  military  context.  He  reviewed  the  specific  issues  the  workshop  was  hoping  to  address  including  some  of  the 
difficulties  in  modelling  complex  human  behaviours: 

1)  Balanced  human  factors  input  to  operational  studies; 

2)  Generation  of  likely  behaviours  and  selection  of  behaviour; 

3)  How  to  tailor  models  fit  for  purpose; 

4)  Performance  metrics; 

5)  Reduction  of  complexity; 

6)  Reduction  of  effort;  and 

7)  Reuse  of  developments. 

He  emphasized  the  need  to  develop  the  right  model  for  the  operator,  at  the  right  time,  for  the  right  cost. 

Dr.  Foster’s  keynote  address  focused  on  the  challenges  facing  modelling  and  Human  behavioural  Representation 
given  the  broad  range  of  scope,  timelines,  and  complexity  of  tactical  and  strategic  operations  that  are  now 
undertaken  by  NATO  countries  still  dealing  with  conventional  and  asymmetrical  adversaries  in  the  form  of 
insurgency  and  terrorist  operations,  but  more  and  more  addressing  issues  of  the  non-kinetic  battlespace  with  its 
complex  socio-political  issues,  and  interactions  between  military  and  civilian  organizations  who  are  dealing  with 
the  full  range  of  security,  stability,  transition,  and  reconstruction  (SSTR)  issues.  He  also  discussed  the  need  for 
Human  behavioural  Representation  to: 

1)  Extend  beyond  experimentation  into  areas  of  training,  mission  rehearsal,  planning,  and  forecasting; 

2)  Expand  the  range  of  scenario  and  behaviour  authorship  to  the  user  of  modelling  tools; 

3)  Incorporate  the  physiology  of  the  operator  rather  than  rely  exclusively  on  empirically  derived 
performance  moderators; 

4)  Include  complex  social  and  cultural  dynamics; 

5)  Develop  better  validation  and  verification  approaches;  and 

6)  Reduce  the  cost  and  time  to  create  constructive  simulations. 

Dr.  Foster  identified  three  technologies  that  could  assist  this  research  and  development:  advances  in  gaming 
technology,  supercomputing  capability  through  cluster/grid  based  technology,  and  the  development  of  open 
databases  of  real-world  data  to  assist  model  development  and  validation. 


12.4  SESSION  SUMMARIES 

12.4.1  Topic  1  -  What  Human  Factors  Does  the  Operation  Involve? 

Dr.  Castor  lead  a  discussion  that  addressed  the  problem  of  determining  the  level  of  fidelity,  abstraction,  and 
simplification  of  the  real  world  that  is  appropriate  for  the  modelling  environment  and  for  the  particular  model 
that  is  required.  A  major  issue  is  determining  what  human  factor  issues  should  be  incorporated  into  these 
simplified  representations  when  dealing  with  the  conflict  between  client  needs  and  available  modelling 
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resources.  A  number  of  other  issues  were  discussed  including  the  lack  of  real-world  data  for  modelling  and 
modelling  validation,  and  the  need  for  highly  trained,  experienced  modellers  with  multiple  domain 
knowledge. 

12.4.2  Topic  2  -  Human  Task  Representation  in  M&S 

Dr.  Allender  lead  off  the  discussion  with  a  review  of  the  types  of  task  analyses  that  are  done  on  real-world 
systems  to  assist  in  the  modelling  environment,  and  outlined  some  of  the  issues  in  the  representation  of  tasks: 
level  of  granularity  in  describing  tasks,  role  of  context  in  defining  tasks,  defining  tasks  for  teams  and  large 
scale  organizations,  and  the  importance  of  natural  language  processing  tasks.  Dr.  Vartanian’s  presentation  on 
unconscious,  naturalistic  decision  making  and  task  performance  in  complex,  rapidly  evolving  situations,  lead 
to  an  extended  discussion  on  the  problems  of  incorporating  such  complex  tasks,  and  the  type  of  unconscious 
decision  making  involved  in  doing  these  task,  into  architectures  for  Human  Behaviour  Representations  that 
have  traditionally  focused  on  rule-based  approaches.  The  issues  and  difficulties  involved  in  incorporating 
emotion  and  creativity  were  also  discussed. 

12.4.3  Topic  3  -  Behaviour  Generation:  Variability  and  Choice 

Mr.  Armstrong  introduced  the  session  by  discussing  some  of  the  requirements  of  Human  behavioural 
Representation,  citing  the  need  for  models  that  are  more  complex,  flexible,  plausible,  and  capable  of  extension 
and  learning  to  supported  unscripted  behaviours,  as  well  as  demonstrate  the  wide  variability  of  individual  and 
team  behaviours.  Two  presentations  were  given  by  Mrs.  Cooper-Chapman  and  Mr.  Emiel  on  modelling 
approaches  that  incorporate  some  of  the  concepts  of  variability  and  choice.  The  UK  Combat  System  Engineering 
Model  focuses  on  situational  awareness,  where  the  experience  of  the  operator  is  a  main  modulator  of  behaviour, 
and  the  TNO  model  incorporates  the  impact  of  the  external  state  of  the  scenario  and  the  internal  state  of  the 
operator  on  generating  stress,  along  with  the  specifics  of  the  task/mission  to  determine  overall  operator 
behaviour.  Following  the  presentations  there  was  extensive  discussion  on  the  approaches  to  use  in  large  scale 
simulations  and  the  inherent  difficulties  in  bounding  the  variability  within  the  behavioural  representations.  It  was 
emphasized  that  the  skill  and  experience  of  the  modeller  is  a  critical  factor  in  developing  appropriate 
simulations. 

12.4.4  Day  2  -  Keynote  Address 

Mr.  Mike  Greenley’s  Keynote  address  discussed  the  Human  behavioural  Representation  issues  from  an 
industry  perspective.  He  noted  several  major  trends  in  the  development  of  M&S  technologies,  including  more 
commercialization  of  the  tools  and  delivery  of  complete  turnkey  simulation  environments  to  the  end-user  with 
continuous  ongoing  facility  and  simulation  support  by  the  contractors  and  the  move  to  simulations  involving  a 
broader  range  of  military  and  non-military  activities,  including  complex  multi-person  teams,  groups, 
and  organizations.  He  stressed  that  more  and  more  intelligent  agent  technologies  will  be  used  in  the  computer 
generated  forces  of  the  large  scale  simulations  and  identified  the  increasing  role  of  the  gaming  industry  and 
the  advances  in  gaming  technology  in  constructive  and  human-in-the-loop  simulations.  There  was  discussion 
following  Mr.  Greenley’s  address  on  the  issues  of  surrounding  the  rapid  delivery  of  simulation  environments 
and  the  role  of  intelligent  agents  in  accelerating  the  development  of  training  capabilities. 

12.4.5  Topic  4  -  Militarily  Relevant  Mental  Output  Measures 

Dr.  Belyavin  lead  an  extensive  discussion  on  the  metrics  of  Workload  (WL)  and  Situational  Awareness  (SA) 
focused  on  seven  questions: 
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1)  Is  Workload  a  well-defined  construct? 

2)  Do  we  need  to  model  it  and  if-so  how? 

3)  Do  we  need  to  validate  workload  predictions  from  models? 

4)  Is  SA  a  well-defined  construct? 

5)  Can  we  validate  measures  of  SA? 

6)  Is  that  a  useful  activity? 

7)  Are  there  other  measures? 

It  was  emphasized  that  there  is  often  poor  correlation  of  the  metrics  with  task-performance,  as  well  as  the 
often  poor  correlation  between  the  metrics  generated  by  the  cognitive  architectures  used  for  system  models 
and  the  metrics  obtained  from  the  operators  of  the  actual  systems.  There  was  agreement  that  most  model 
architectures/environments  are  not  designed  to  produce  estimates  of  the  metrics,  especially  SA,  as  they  focus 
on  performance  prediction.  Dr.  Allender  supported  the  need  for  reporting  SA  and  WL  metrics,  as  it  provides 
the  client  with  valuable  information.  There  was  a  general  consensus  that  the  WL  and  SA  constructs  cannot  be 
discarded,  but  more  work  is  needed  to  better  define  and  validate  the  SA  construct. 

12.4.6  Topic  5  -  The  Concept  of  Moderators 

Dr.  Belyavin  also  led  a  wide-ranging  discussion  on  the  moderators  of  human  behaviour  focusing  on  four 
specific  questions: 

1)  For  what  applications  do  we  need  to  solve  the  moderator  problem? 

2)  Can  we  ignore  moderators  across  a  range  of  problems? 

3)  How  do  we  develop  valid  models  if  we  need  to? 

4)  How  do  we  deal  with  potential  moderator  interactions? 

Though  the  impact  of  external  stresses  such  as  the  environment  are  recognized  as  key  moderators,  much  more 
extensive  research  and  data  is  needed  on  other  moderators,  especially  on  the  issues  of: 

1)  Moderator  characterization; 

2)  The  interaction  of  moderators  such  as  experience  and  personality  type  with  other  moderators  such  as 
the  environment;  and 

3)  Traditional  and  new  definitions  of  moderators  in  the  context  of  teams,  organizations,  and  complex 
military  and  civilian  operations. 

It  was  emphasized  that  there  is  no  simple  “state”  description  of  human  capability.  Rather,  “state”  is  a  complex 
function  of  task,  training,  personality,  environment,  etc. 

12.4.7  Topic  6  -  Complexity,  Hierarchy,  Modularity  and  Validity  in  HBR  Architectures 

Dr.  Gluck  introduced  some  of  the  issues  surrounding  the  design  and  structure  of  the  currently  used  cognitive 
architectures  such  as  ACT-R  and  SOAR,  emphasizing  the  slow  pace  of  development  of  cognitive  architectures 
and  modelling  environments.  Dr.  Jones’  presentation  on  Comparative  Analysis  of  Frameworks  for  Knowledge- 
Intensive  Intelligent  Agents,  emphasized  the  lack  of  modularity  of  the  architectures  which  would  allow  better 
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reuse  of  model  components,  the  difficulty  in  separating  theory  and  implementation  in  the  architectures,  the 
inability  to  transfer  cognitive  models  between  architectures,  and  the  difficulties  in  adding  intelligent  agent 
capabilities  to  the  current  architectures.  He  emphasized  the  need  to  follow  a  rigorous  software  engineering 
approach  in  building  new  architectures  for  intelligent  behaviour  that  address  understandability  and  usability 
challenges  from  the  outset  and  which  would  include  comprehensive  support  for  necessary  knowledge 
representations,  and  well-defined  software  components.  He  also  outlined  a  long  term  strategy  to  meet  these 
goals,  that  included  the  development  of  an  abstract  machine  based  on  common  functional  components, 
interoperable  object  libraries  as  instantiations  of  abstract  components,  a  formal  framework  to  provide  a  bridge 
between  science  and  implementation,  and  an  ability  to  compose  components  quickly  into  the  “best”  architecture 
for  a  given  task. 

Dr.  West’s  presentation  examined  the  different  ways  of  looking  at  a  model’s  validity  including  face,  model, 
predictive,  architectural,  and  cross-model  comparative  validity  and  also  emphasized  the  need  to  link  the 
models  back  to  the  neural  architecture  of  the  brain.  He  also  raised  the  difficulties  of  publishing  and  sharing  the 
models.  The  presentations  were  followed  by  an  animated  discussion  on  how  to  address  the  shortcomings  of 
the  current  architectures.  There  appeared  to  no  consensus  on  how  to  proceed,  though  the  current  restrictions 
on  model  development  and  sharing  were  recognized. 

12.4.8  Topic  7  -  From  Individual  to  Group  Behaviour 

Mr.  Cain  opened  the  session  with  a  talk  on  the  complexities  and  difficulties  in  extending  models  of  Human 
behaviour  to  include  group  behaviours.  He  proposed  three  problem/discussion  areas: 

1)  Can  we  define  those  essential  attributes  of  teams  that  require  formal  models  that  would  make  team 
entity  modelling  viable  and  are  validated  formal  models  available? 

2)  Can  we  model  teams  of  individuals  at  sufficient  resolution  and  validate  them  for  use  in  military 
simulations  and  what  can  make  this  process  affordable? 

3)  Can  we  define  the  pros  and  cons  of  each  approach  sufficiently  so  that  the  military  M&S  community 
can  make  an  informed  decision  about  which  is  more  appropriate  in  a  given  context? 

4)  How  is  an  organization  representation  different  from  a  small  team  and  do  we  need  to  represent 
explicit  EBO  behaviours  or  is  it  sufficient  to  represent  their  “effects”? 

Modelling  teams  as  both  individuals  and  collectives  can  be  appropriate,  however  models  of  team  leaders  or 
intelligent  agents  acting  to  coordinate  activities  is  a  critical  modelling  issue.  There  is  active  research  in  crowd 
modelling,  and  socio-cultural  team  modelling,  but  model  approaches  for  complex,  non-kinetic  mixes  of  military 
and  civilian  organizations  is  lacking. 


12.5  COMMENTS  AND  RECOMMENDATIONS 

The  structure  of  the  meeting,  with  the  focus  on  modelling  issues  rather  than  presentations  on  specific 
modelling  tools  or  models,  provided  the  forum  for  a  wide-ranging  discussion  of  the  many  and  complex 
problems  and  challenges  faced  by  those  involved  in  Human  behaviour  Representation  in  constructive 
modelling  and  simulation.  One  of  the  central  themes  arising  from  both  Keynote  addresses  is  the  increasing 
need  to  incorporate  a  wide-range  of  Human  behavioural  Representations  into  constructive  simulations  of 
complex,  non-kinetic,  operations  involving  a  mix  of  military  and  civilian  organizations  (both  government  and 
NGOs)  and  civilian  populations,  taking  into  account  broad  socio-political  issues.  It  is  also  apparent  that  while 
more  work  is  required  on  developing  individual  and  team  models,  there  is  a  serious  deficiency  in  tools  for 
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advanced  model  development,  model  sharing,  simulation,  and  validation  -  key  factors  in  the  development  of 
useful  simulations  of  large,  complex  systems.  Modelling  software  technologies  and  database  capabilities  must 
be  advanced  to  assist  the  community  in  rapidly  developing,  testing,  validating,  sharing,  and  using  human 
behaviour  and  performance  models  that  can  be  integrated  into  the  full  spectrum  of  M&S  capabilities.  It  is  also 
apparent  that  a  generic  component  based  approach  to  Human  behavioural  Representation  is  required  that  will 
facilitate  a  plug-and-play  approach  to  implementing  human  models  in  large,  complex  constructive  and  HITL 
simulations  involving  system-of-systems.  This  effort  will  require  the  development  of  new  modelling  tools, 
as  well  as  the  embedding  of  HLA  and  other  interfacing  capabilities  into  existing  technologies.  NATO  should 
focus  less  on  human  based  experimentation,  more  on  real-world  data  collection  and  the  development  of  open- 
access  databases,  and  more  on  development  of  general  behavioural  models  that  can  be  incorporated  into  a 
broad  range  of  constructive  simulations. 

Though  there  is  a  desire  to  move  authorship  of  simulations  more  and  more  to  the  client  organizations,  it  must 
be  recognized  that  the  validity  of  the  models  are  highly  dependent  on  the  training,  skill,  and  experience  of  the 
modelers,  who  require  domain  knowledge  in  modelling  theories  and  methodologies,  in  specialized  software 
and  programming  languages,  in  mathematics  and  statistics,  as  well  as  in  all  aspects  of  the  real-world  situation 
they  are  attempting  to  simulate.  When  dealing  with  complex  system-of-systems,  the  multi-domain  knowledge 
requirements  can  be  overwhelming.  In  spite  of  the  advances  in  computing  power,  developing  and  using 
advanced  Human  behaviour  Representations  will  remain  an  expensive  and  labor  intensive  discipline. 

In  order  to  enhance  the  development  of  Human  behavioural  Representations,  and  facilitate  their  incorporation 
into  the  larger  M&S  capabilities,  NATO  RTO  should  consider  the  establishment  of  several  Working  Groups 
to  focus  on  specific  raised  by  the  Specialists’  Meeting. 

NATO  RTO  should  initiate  a  series  of  initiatives  to  address  the  development  of  advanced  modelling  tools  to 
expand  the  capabilities  in  simulating  individuals,  individuals  within  teams  and  organizations,  and  teams  and 
organizations.  This  will  require  the  integration  of  models  of  emotion  and  creativity  and  more  emphasis  on 
biological/physiological  bottom-up  models  of  brain  function.  One  possible  approach  is  develop  human 
behaviour  and  performance  models,  using  an  open-source  modelling  tool  such  as  Modelica,  specifically 
designed  for  multi-domain,  system-of-systems  modelling,  capable  of  handling  a  mix  of  mathematical 
representations,  and  allowing  for  the  integrated  modelling  of  humans,  hardware,  and  communication 
components,  as  well  as  complex  socio-political-economic  systems  This  effort  should  also  focus  on  developing 
tools  for  sharing  models  among  different  authoring  tools  by  the  development  of  XML  schemas  for  model 
description  and  developing  software  to  link  multiple  modelling  technologies  including  Modelica,  IPME, 
IMPRINT,  SOAR,  ACT-R,  fuzzy-logic  rule  based  systems,  and  neurophysiological  based  modelling 
environments  such  as  PDP++/leabra++  . 

In  order  to  develop  and  validate  performance  and  behavioural  models,  simulations  and  real-world 
experimentation  there  is  a  need  to  use  well  defined  metrics,  beyond  the  current  individual  Workload  and 
Situational  Awareness  measures,  which  are  difficult  to  define  in  individuals,  and  poorly  defined  in  terms 
teams,  groups,  and  networked  operations.  NATO  RTO  should  establish  Working  Groups  to  develop  new 
approaches  to  performance  and  behaviour  metrics  in  large  scale  constructive  simulations  and  real-world 
operations.  Given  the  shear  complexity  of  the  large  scale  simulations  of  military  and  large  scale  civilian 
operations,  this  should  include  the  development  of  technologies,  including  intelligent  agents,  for  large  scale 
automated  data  reduction,  analysis,  and  display,  utilizing  advances  in  GRID  technologies. 

In  parallel  to  the  work  on  simulation,  experiment,  and  real-world  metrics,  NATO  should  establish  a  Working 
Group  on  the  issues  of  data  collection  and  data  storage,  to  handle  the  immense  volume  of  data  collected  on 
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human  behaviour  and  performance.  In  addition  to  the  issues  surrounding  raw  data  storage,  the  Working  Group 
should  address  database  methodologies  for  storing  information  on  data  collection  methodologies  used, 
data  format  and  data  conversion  techniques,  calibration  procedures,  and  model  databases. 
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Chapter  13  -  CONCLUSIONS 


13.1  SUMMARY 

The  two  days  of  collegial  discussions  on  Human  Behavioural  Representation  issues  seemed  to  be  a  successful 
forum  for  the  exchange  of  ideas  and  exploration  of  concepts.  The  focus  on  plenary  discussion  with  short 
presentations  seemed  to  be  well  received  and  there  was  extensive  participation  in  the  discussions  by  all  of  the 
attendees. 

There  was  general  agreement  with  the  points  of  view  of  HFM-128  approach  to  documenting  the  human 
factors  of  HBR  and  that  the  community  seems  to  be  on  the  right  track.  Although  progress  in  modelling  human 
factors  has  been  slow  over  the  past  decade,  other  forums  such  as  BRIMS1  and  the  CogSci2  conferences  have 
been  reporting  a  number  of  theoretical  and  applied  papers  on  human  behaviour  and  performance  modelling. 
While  we  are  still  a  long  way  from  turnkey  models  of  operators,  the  consensus  that  various  modelling 
approaches  are  useful  gives  confidence  to  move  ahead  in  the  field  from  a  variety  of  perspectives. 

Situation  awareness  and  workload  are  concepts  that  are  intuitive  to  the  military  client,  yet  they  are  not 
necessarily  useful  as  a  scientific  concept  that  should  be  modelled.  However,  metrics  of  such  concepts  may  be 
generated  by  models  to  assist  interpretation  of  the  model’s  performance  predictions. 

A  modular  approach  to  modelling  human  activity  seems  to  be  the  norm,  lending  itself  to  inclusion  of  moderators 
while  attempting  to  simplify  the  complex  field  of  human  cognition.  The  human  cognitive  and  performance 
modules  benefit  from  an  abstraction  layer  that  is  an  interface  between  the  modules  and  the  modelled 
environment,  allowing  models  to  be  reused  across  similar  applications,  requiring  principally  minor  changes  to 
the  interface  layer. 

When  going  from  individual  HBRs  to  team  and  larger  unit  characteristics,  one  cannot  simply  link  the  models 
and  expect  to  get  plausible  behaviour.  There  are  additional  behaviours  and  goals  that  characterize  groups  that 
need  to  be  added  to  the  model  knowledge  base. 


1  Behavior  Representation  in  Modeling  and  Simulation:  http://www.sisostds.org/. 

2 

Cognitive  Science  Society:  http://www.cognitivesciencesociety.org/cogsci.html. 
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13.2  RECOMMENDATIONS  OF  HFM-143  SPECIALISTS’  MEETING  ON  HER  IN 
CONSTRUCTIVE  SIMULATION 


General 

To  R&D  Community 

To  NATO 

1 

Develop  a  capability  to  model 
non-traditional  military  activities 
such  as  humanitarian  aid  and 
state  reconstruction.  These 
activities  are  increasingly 
becoming  the  mandate  of  the 
military,  OGD  and  NGO  as  a 
result  of  the  3-Block  war 
concept  and  Effects  Based 
Operations  with  the  intent  of 
predicting  group  and  population 
reactions  to  events. 

Develop  formal  modelling 
techniques  to  represent  effects 
on  organizations  and  populations 
as  well  as  individuals  that  allow 
for  generation  and  analysis  of 
chaotic  behaviours  (small  cause, 
large  consequences). 

Hold  a  joint  meeting  of  SAS- 
053,  HFM-128  and  other 
interested  parties  to  look  for 
synergies  arising  from 
collaboration  on  HBR  issues  in 
NATO  M&S  with  the  intent  of 
establishing  a  multidisciplinary 
panel  to  define  the  scope  and 
requirements  for  constructive 
modelling  of  Effects  Based 
Operations,  including  members 
from  SAS  and  HFM  currently 
working  on  HBR. 

2 

Advance  the  state  of  the  art  in 
HBR  such  that  analysts  can  self¬ 
author  behavioural  models  for 
CGF/SAFs. 

Develop  task  representations 
that  are  compact  in  specification 
and  show  behavioural  richness 
both  by  reactivity  and 
proactivity.  Continue  the  trend 
toward  modular  HBR  systems 
and  cognitive  architectures  to 
promote  reuse  and  ease  of 
refinement  of  models. 

3 

Educate  stakeholders  in  proper 
use  of  study  design  and  analysis. 

More  exchanges  between 
military  customers  of  HBR  and 
modellers  to  get  an 
understanding  of  what  is 
required  from  both  perspectives. 

Clearly  establish  the  constraints 
and  approximations  used  in  a 
model  to  determine  the  range  of 
applicability,  or  its  accreditation 
for  use. 

Organize  a  meeting  in  which 
military  M&S  stakeholders 
collaborate  to  integrate  human 
factors  into  simulations. 

Develop  a  NATO  checklist  of 
human  factors  and  document 
why  each  factor  should  be 
included  as  a  means  good 
practice,  extending  this  into  a 
guided  interview  that  will  help 
frame  the  model  space. 

4 

Validate  models  and  document 
their  range  of  applicability. 

Develop  procedural  and 
statistical  methods  of  performing 
validation  of  models.  Note  best 
practices  for  gaining  insight  into 
behaviours  and  performance  as 
well  as  techniques  to  quantify 
goodness-of-fit. 

Collect  and  publish  data  that  can 
be  used  to  build  and  validate 

HBR  suitable  for  military  M&S. 
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General 

To  R&D  Community 

To  NATO 

4 

Establish  public  datasets  from  a 
wide  range  of  tasks  in  order  to 
objectively  establish  the  broader 
validity  of  the  cognitive 
architectures  and  HBR  systems. 

Promote  the  sharing  of  databases 
of  behaviour  and  performance 
observations  that  can  be  used  to 
shape  models  based  on  doctrinal 
procedure  rules  to  promote  more 
plausible  behaviour.  This  could 
possibly  be  done  in  conjunction 
with  the  HBR  Virtual  Institute. 

Provide  a  mechanism  to  capture 
operational  data  that  can  be  used 
to  build  and  validate  higher  level 
models.  Recommend  the  use  of 
the  NATO  training  facilities  to 
collect  data  on  high  level  C2 
processes. 

5 

Develop  a  means  of  introducing 
plausible  variability  into  HBR 
through  the  use  of  learning  and 
the  introduction  of  formal 
models  that  capture  within  and 
between  individual  variability  or 
error. 

Introduce  factors  such  as 
learning  (functional  and 
procedural  relationships), 
experience,  context,  goals  and 
emotions  into  modelling 
operator  performance. 

Define  the  different  operator 
parameters  that  need  to  be 
considered  for  the  modelling  of 
performance  and  behaviour 
common  for  military  operations. 
Define  how  these  relate  to  the 
moderators  of  interest.  Split 
sources  of  performance  variation 
in  traits  (moderators,  constant 
during  the  simulation  period) 
and  states  (dynamic  outcome 
based  on  state  variables). 
Investigate  if  a  similar  approach 
to  the  successful  use  of  state 
variables  in  physiological 
performance  can  be  applied  to 
states  in  the  cognitive  and 
emotional  domains. 

Assemble  and  make  public  an 
electronic  library  of  moderator 
models  and  their  effects  on 
performance  (preferably  open 
source)  that  analysts  can  use  in 
HBRs,  providing  validation 
details  and  suitable  application 
of  the  models,  possibly  in 
conjunction  with  the  HBR 

Virtual  Institute. 
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General 

To  R&D  Community 

To  NATO 

5 

Develop  or  document  formal 
models  of  emotion  and 
motivation  based  on  models 
from  the  social  sciences  that  can 
be  integrated  within  current 

HBR  architectures. 

6 

Development  of  an  open  source 
modelling  environment  that 
could  evolve  into  an  industry 
standard  for  the  interface  of 

HBR  to  other  simulations. 

Define  the  architectural 
constraints  for  implementation 
of  state  variables  and 
performance  shaping  functions 
in  the  cognitive  and  emotional 
domains. 

Develop  middleware  that 
facilitates  cross-HBR 
comparison.  The  HBR  R&D 
community  would  likely  benefit 
from  a  better  understanding  of 
productive  methods  for 
comparing  models  across 
architectures  and  comparing 
architectures. 

Development  of  a  hierarchical 
human  HLA  FOM  or  a  similar 
interface  specification  that  meets 
the  foreseeable  requirements  of 
NATO’s  M&S  activities. 

Document  modelling  synthetic 
environments,  HBR 
architectures  and  models  that 
currently  support  non-kinetic 
warfare,  including  social- 
cultural  effects  typical  of  post¬ 
combat  reconstruction,  noting 
extensions  that  would  be 
required  to  make  other  products 
currently  in  use  applicable  to 
cost  effective  EBO  simulations. 

7 

Improve  the  HBR  element  of 
team  and  group  models. 

Develop  guidance  on  suitable 
means  of  modelling  the  way 
individual  behaviour  should  be 
aggregated  at  team  and  larger 
unit  level. 

Document  descriptive, 
conceptual  and  formal  models 
that  attempt  to  capture  the 
effects  of  team  and  crowd  latent 
variables  on  behaviour.  Develop 
guidance  to  the  M&S 
community  on  the 
appropriateness  of  various  team 
and  crowd  modelling  approaches 
to  application  areas. 

Explore  how  task-analytical 
techniques  can  be  extended  to 
include  teamwork  to  support 
modelling  teams  as  collections 
of  individuals. 

Build  upon  NATO/RTO  panel 
studies  on  teams  and  C2 
modelling  to  identify  formal 
models  of  teams  and  crowds  as 
well  as  the  latent  factors  that 
characterize  groups. 
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9:45 
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Dr.Bob  Foster 

10:15 

Coffee  Break 

10:30 

Topic  1 :  What  human  factors  does  the  operation  involve? 

Martin  Castor 

Andy  Belyavin 

12:00 

Lunch  -  Officers'  Mess 

13:00 

Topic  2:  Human  task  representation  in  M&S 

Laurel  Allender 

Dr.  Oshin  Vartanian 

14:30 

Coffee  Break 
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Topic  3:  Behaviour  generation  -  variability  and  choice 

Joe  Armstrong 

Mrs.  Carol  Cooper- 
Chapman 

Mr.  Emiel  Ubink 
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Bill  Fraser 
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Wrap  up 

Brad  Cain 
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Mr. Mike  Greenley 
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Andy  Belyavin 

Joe  Armstrong 
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Andy  Belyavin 
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Dr.  Randy  Jones 
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Dr.  Rob  West 
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Stretch 

16:20 

Summary 

Bill  Fraser 
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Brad  Cain  &  Wouter  Lotens 
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Appendix  3  -  PRESENTATION  ST  IPES 
A3.1  HFM-128  CHAIRMAN’S  MESSAGE 


RTO  HFM143  Specialist 
Meeting 


Human  Behavior  Representation 
in  Constructive  Modeling 


Purpose 


•  To  discuss  recommendations  to  NATO 
regarding  simulation  of  human  behavior  in  a 
military  context 

•  based  on  selected  topics  at  the  core  of  behavior 
generation 

•  while  bringing  HF  and  operational  field  together 
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HFM  128 -HFM  143 


•  HFM  128  on  HBR  in  constructive  modeling  to 
report  in  concept  this  summer 

•  Concept  report  is  input  to  HFM  143.  A  separate 
report  will  be  produced  on  the  Specialist  Meeting 


Still  a  problem  .... 


•  Balanced  HF  input  to  operational  studies 

•  Generation  of  likely  behaviors  and  selection  of 
behavior 

•  Models  fit  for  purpose 

•  Performance  metrics 

•  Reduction  of  complexity,  reduction  of  effort,  reuse  of 
developments 
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HFM  128  vision 


•  Careful  framing  of  a  study 

•  Science  based  parameter  choices 

•  Moderator  concept  to  reduce  complexity  of  HF 

•  Behavioral  choices  to  generate  course  of  action 

•  Performance  in  military  perspective 


This  meeting 


•  Selected  topics 

•  Discussions  introduced  and  steered  by  HFM  128  reps 

•  Notes  and  reporting. 

•  Your  involvement  is  key 
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A3.2  KEYNOTE  ADDRESS  -  DAY  1 


NA  TO  Workshop:  Representation  of 
Human  Behavior  in  Constructive 
Simulation 

Dr.  Robert  Foster 
Director,  BioSystems 

(Office  of  the  Director,  Defense  Research  and  Engineering) 


A3 -4 


RTO-MP-HFM-1 43 


APPENDIX  3  -  PRESENTATION  SLIDES 


3 


RTO-MP-HFM-1 43 


A3 -5 


APPENDIX  3  -  PRESENTATION  SLIDES 


ORGANIZATION 


main  Space  and  M&S  Applications 


Irregular 

SSTR 


Conventional- 


L» 


B 

Months 


■■b/  Minutes 


Modeling  Human  Behavior 


Where  is  the  current  State  of  the  Art? 

-  Underlying  science  and  theory 

•  Cognitive  Theories/Modeling 

•  Human  Performance  Modeling 

•  Physiological  Systems 

•  Emotional,  Social  and  Cultural  Theories/Models. 

-  Supporting  Technologies 

•  High  Performance  Computing 

•  Commercial  Technologies 

-  Challenges 

•  Affordability 

•  Flexibility 

•  Ease  of  Authoring 

•  Flexibility 

•  Verification 

•  Validation 
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Cognitive  Science  and  Theories: 

-  The  ‘Decade  of  the  Mind’ 

•  DoD  cognitive  science  investments  are  ‘stable’  in 

-  Human-Computer  Interface  (Future  Combat  Systems,  Helmet 
Mounted  Display  Technologies  within  Joint  Strike  Fighter,  Navy  DDG- 
1000) 

-  Training  and  Simulation  (Leadership,  Socio-cultural  awareness,  etc.) 

•  DoD  investment  in  socio-cultural  understanding  and  modeling  is 
increasing 

-  Recognition  of  need  for  broader  understanding  of  socio-cultural 
factors  from  Phase  0  to  Phase  4  of  military  operations 

-  Security,  Stability,  Transition  and  Reconstruction  (SSTR)  operations 

-  New  Start  program  in  Human,  Social,  Culture  and  Behavior  Modeling 


Focus  is  on  transition  of  cognitive  &  behavioral  models 
into  systems/programs  of  record. 

-  Need  to  be  extensible  beyond  ‘Experimentation’ 

-  Training  Systems,  Mission  Rehearsal,  Planning,  Forecasting 

We  can’t  have  the  modelers  doing  the  ‘modeling’ 

-  To  be  successful,  end  user  authoring  tools  must  be  developed 

-  Must  allow  users  to  create  entity  behaviors,  not  just  scenarios 

-  Need  to  be  able  to  understand,  model  and  code  general 
principles  of  behavior 

-  Must  decrease  the  manhours  and  cost  to  create  simulations 
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Verification,  Validation  and  Credibility 


Validation 


Credibility 


Verification  &  Validation 


Verification 

-  There  is  a  proliferation  of  models  and  simulations  that  go  straight  to  use  without 
V  V  &  A 

-  Lack  of  proper  documentation 

-  Lack  of  empirical  analysis 

Validation 

-  Face  validity  vs.  Structural/Predictive  validity 

•  Going  beyond  SME-based  assessments 

•  Going  beyond  the  existing  range  of  parameters 

•  Many  models  are  not  empirically  testable 

-  Structure  vs.  Content 

•  Frameworks  and  architectures  often  come  first 

•  Some  frameworks  can  be  used  as  testbeds  for  demonstration  and 
validation 

Credibility 

-  When  the  real  world  is  poorly  understood... then  the  conceptual  model  is  poor 
and  it  may  be  verified,  not  valid  AND  not  credible 

-  Need  for  effectiveness  measures 
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There  Are  Some  Successes,  I  Think 


Training  &  Mission  Rehearsal 

Simulation  Infrastructure 
for  Mission  Rehearsal 


Authoring  Tools 


After-Action  Review  & 
Training  Effectiveness 


V_i_  ^ 

Common  Distributed  Mission  Training  System  (C-DMTS) 


Training  in  Operational  Platforms 


r  Navy  ASW 

Navy  Weapons 

,  IMAT 

VIRTE  , 
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Are  There  Testing  Environments  ? 


Traditional  M&S 

•  SEAS 

.  77 


Games 

•  RealWorld 

•  “A  Force  More  Powerful”  (BreakAway  Ltd) 

.  77 


=  13  = 


In  Summary  There  Are  Challenges 

•  Authoring 

□  From  real  experiences 

□  Application  Users  (not  experts/” pucksters”) 

•  Behaviors 

□  Include  actual  physiology  (not  just  “moderators) 

□  Include  human  social  &  culture  dynamics 

•  Validation 

□  Data 

□  Datasets  for  research  and  developers 
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A3.3  WHAT  HUMAN  FACTORS  DOES  THE  OPERATION  INVOLVE? 


What  human  factors  does  the 
operation  involve 

Martin  Castor 

Swedish  Defence  Research  Agency 


Basics 


•  We  are  modeling  something  very  complex 

•  Modeling  is  about  abstraction 

•  A  lot  of  abstractions  necessary  in  HBR 

•  Central  issue  of  any  modeling  project 
decide  which  are  the  relevant  factors 

•  Which  human  factors  does  the  operation 
involve? 


i 
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”REWA  Worlds” 


The 

"Real  World" 


The 

"Problem  World" 


The 

"M&S  World" 


The 

"Product  World" 


Objectives 

Needs 


Decisions 

Actions 


Problem 

Formalisation 


i 


Problem 

solution 


M&S 

Questions 


i 


M&S  Answers 


* 


M&S  System 
Requirements 


M&S  Enabling 
Products 


Picture  from  REWA  2  project  .To  appear  on  SISO  General  Methodology  WA  this  summer 


ft 


|  NATO 
OTAN 


ORGANIZATION 


Discussion 


It  depends! 


We  can  model  “everything”  but  conceptual 
model  and  empirical  data  often  a  problem 

Can  formal  approaches  to  selection  of 
relevant  human  factors  and  scenarios 
exist? 

Advice  to  NATO 


NATO 


OTAN 


ORGANIZATION 
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Questions 


•  How  to  link  customer  higher  level 
questions,  e.g.  sustainability  of  force,  to 
requirements  for  which  HF  to  include? 

•  How  to  choose  the  appropriate  fidelity 
level  for  representation? 

•  How  to  do  trade-offs  between  level  of 
detail  and  budget? 


Questions 


•  Traceability  and  explainability  of  behavior 
are  very  important,  but  does  this  lead  us  to 
choose  too  simple  scenarios? 


El 
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m 


Representing  human  factors 

Choice  of  what  human  factors  need  to  be 
represented  is  an  important  decision  in 
any  study 

By  way  of  illustration  I  will  describe  a  study 
involving  a  complex  two  part  model 

An  early  decision  was  made  to  limit  the 
human  factors  detail 

Was  it  right? 


^NATO 
OTAN 


ORGANIZATION 


Study  Objectives 

Basic  principle  operated  in  the  UK  low 
flying  system  (LFA)  is  “See  and  Avoid” 

Responsibility  lies  with  aircraft  flying  in  the 
system  to  detect  potential  conflicts  visually 
and  take  appropriate  action 

Primary  aim  of  the  study  was  to  identify 
the  effect  of  measures  that  could  reduce 
random  collision  rate 


jJBnato 
fP  OTAN 
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Approach  adopted 


•  Clearly  human  factors  in  collision  avoidance 

-  Visual  detection 

-  Avoidance  action 

•  Possible  human  factors  in  flight  tracks... . 

•  Since  intersections  between  tracks  are  random 
decided  that  no  need  to  describe  HF  in  flight 
pattern 

•  Could  split  the  model  into  two  parts: 

-  Track  &  intersection  generation 

-  Given  track  geometry  can  the  potential  decision  be 
detected? 


Track  generation 

•  Track  generation  is  not  simply  set  of  tracks 
in  arbitrary  directions 

•  Intersections  are  a  “squared”  problem 


(n*n) 


•  Low  flying  area  has  some  flow  control 
influencing  local  density 

•  Traffic  (Military  &  Civilian)  has  different 
altitude  patterns 
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Generation  Model 

•  Model  constructed  of  traffic  in  LFA 
according  to  observed  data  on  patterns 

•  Careful  construction  so  that  traffic  density 
would  conform  to  observations 

•  Generate  a  series  of  potential  conflicts 

•  Define  the  geometry  of  the  conflict  in 
terms  of  directions/angles/aircraft  types 

•  Validate  conflict  rate  against  observations 


B 


Validation  basic  model 

•  For  Military/Military  and  Civil/Civil  conflict 
rate  validated  satisfactorily  against 
reported  air  prox.  Reports 

•  Military/Civilian  was  not  consistent  with 
observations  or  with  the  other  categories 

•  Indications  of  over-reporting 
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Collision  geometry 


Visual  detection  of  a  potential  collision 
depends  on  4  aspects 
-Target  contrast 
-Target  size 

-  Effective  approach  rate 

-  Number  of  observers 


I  NATO 
OTAN 


ORGANIZATION 


A3 -18 


RTO-MP-HFM-1 43 


APPENDIX  3  -  PRESENTATION  SLIDES 


Visual  detection 


•  Visual  detection  further  depends  on 
-Time  of  day 

-  Day  of  year 

-  Geometry  relative  to  solar  position 


B 


Full  model 


•  Generate  conflicts  using  first  model  run  for 
1 00  years 

•  Use  HF  visual  detection  model  to  assess 
probability  of  detection  a  prescribed 
number  of  seconds  from  impact 

•  Can  vary  assumptions  about  detection 
aids  and  compare 

•  Use  baseline  to  validate  model 
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1 


Summary 


•  Full  model  validated  at  collision  and 
conflict  level 

•  Composed  of  a  non-HF  piece  and  a  critical 
HF  piece 

•  Overall  model  appeared  satisfactory 

•  Two  components  distinct 


B 
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A3.4  HUMAN  TASK  REPRESENTATION  IN  M&S 


Human  Task  Representation 
in  M&S 

Dr.  Laurel  Allender 
U.S.  Army  Research  Laboratory 


I NATO 
^  OTAN 


ORGANIZATION 


Understanding  the  Tasks 

Mission  Scenario... 

-  The  year  is  2015.  Colonel  Henry  Schmidt  is 
leading  a  NATO  coalition  in  the  conduct  of  a 
humanitarian  effort  amidst  skirmishes  between 
local  factions.  The  delivery  of  medical 
supplies  is  well  underway  when  Schmidt 
receives  an  urgent  message  about  a  possible 
hostage  situation.  He  immediately  realizes  that 
this  implies  a  number  of  steps  that  are 
required  for  his  new  tasking. 


NATO 


OTAN 


ORGANIZATION 
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ORGANIZATION 


Understanding  the  Tasks  for  M&S 

Task  Analysis 

-  Starting  point  for  human  factors  efforts 

-  Military  regulations  describe/perscribe  task 
analysis 

-  Types  of  task  analyses 

•  Cognitive 

•  Goal  directed 

•  Hierarchical 

•  Work-centered... 


m 


^NATO 
OTAN 


■ 


ORGANIZATION 


Understanding  the  Task: 
’’Secure  the  Building” 

The  Army  surrounds  the  building  with  defensive 
fortifications,  tanks,  and  concertina  wire. 

The  Marine  Corps  assaults  the  building  using 
overlapping  fields  of  fire  from  all  appropriate 
points  on  the  perimeter. 

The  Navy  turns  out  the  lights  and  locks  the 
doors. 

And  the  Air  Force  takes  out  a  three-year  lease 
on  the  building  with  an  option  to  buy. 


jJBnato 
fP  OTAN 
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So,  What  Must  Be  Considered  to 
Represent  Human  Tasks  in  M&S? 

What  level  of  task  granularity  or  aggregation  is 
appropriate? 

At  what  level  or  under  what  circumstances  is  it  sufficient 
to  represent  the  end  state  of  task  performance?  Should 
the  underlying  process  that  generated  the  performance 
be  represented? 

How  much  context  must  be  specified? 

Can  we  create  an  engine  that  allows  us  to  give  high-level 
commands  to  the  model? 

Is  natural  language  understanding  a  desired  or 
necessary  capability  for  human  behavior 
representations? 


NATO 


OTAN 


ORGANIZATION 


Task  Representation 
Considerations 


How  a  task  happens 
Method 


|  NATO 
OTAN 


That  a  task  happens 
End  result 


Where  does  context 
specification  fit? 
Learning? 

Natural  lanuage 
commands? 

How  does  this  vary 
for  concept 
exploration,  design, 
training,  or  mission 
rehearsal? 
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Role  of  Conscious  Deliberation  in  Simple  vs.  Complex  Tasks 


Oshin  Vartanian 

Defence  Research  and  Development  Canada 


Defence  Research  and  Recherche  et  developpement 
■  ■  Development  Canada  pour  la  defense  Canada 


Canada 


Problem 


How  is  a  task  (e.g.,  a  command)  represented? 

-  e.g.,  “Secure  the  building!” 

-  Explicitly  vs.  implicitly? 

How  does  conscious  deliberation  affect 
representation? 

-  Conscious  deliberation  =  attention 

Key  Question:  Is  conscious  deliberation  always 
advantageous? 


Defence  R&D  Canada  -  Toronto  •  R&  D  pour  la  defense  Canada  -  Toronto 
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Problem  (cont.) 


•  Classical  wisdom: 

-  Conscious  deliberation!  =>  quality  of  choice  T 

-  Time  pressure/anxiety  =>  quality  of  choice^ 


•  Is  this  true  for  tasks  of  varying  complexity? 
omplexity  =  Number  of  attributes 


Defence  R&D  Canada  -  Toronto  •  R&  D  pom'  la  defense  Canada  -  Toronto 


Problem  (cont.) 


•  Complex  tasks  have  many  attributes 
-  e.g.,  “Secure  the  building!” 

•  Time 

•  Space 

•  Sequencing  of  actions 

•  Cost  and  benefit  of  engagement 

•  etc. 
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1 


ORGANIZATION 


Problem  (cont.) 


•  Key  insight:  Value  of  conscious  deliberation  is  a 
function  of  complexity: 


Simple  tasks  (e.g.,  buying  shoes)  =>  value! 
Complex  tasks  (e.g.,  choosing  a  car)  =>  value! 


^  T 


Dijksterhuis  et  al.  (2006) 


Fig.  1.  Percentage  of  participants  who  chose  the 
most  desirable  car  as  a  function  of  complexity  of 
decision  and  of  mode  Off  thought  \n  =  18  to  22  in 
each  condition).  Error  bars  represent  the  stan¬ 
dard  error. 

Defence  R&D  Canada  -  Toronto  •  R&  D  pom'  la  defense  Canada  -  Toronto 


Problem  (cont.) 


•  Conscious  deliberation: 

-  Limited  capacity 

•  Attention  can  be  directed  at  few  attributes 

-  Rule-based  (e.g.,  don’t  exceed  a  certain  price) 

mscious  (implicit)  deliberation: 
eared  toward  pattern  recognition 
ot  limited  by  capacity 
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Problem  (cont.) 

•  Upshot?  Deliberation  will  not  help  if  task  is 
complex 

•  This  is  true  in  terms  of: 

-  Objective  quality  of  choice 
—  Post-choice  satisfaction 


RO7  Implications:  Operations 


Training  is  key! 

-  Must  facilitate  pattern  recognition. 

-  Command  carried  out  without  deliberation. 


Defence  R&D  Canada  -  Toronto  •  R&  D  pour  la  defense  Canada  -  Toronto 
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ruy  Implications:  M&S 


Emotion? 

-  Negative  emotion  (anxiety)  =>  ^cognitive  capacity 

-  ^cognitive  capacity  =>  ^conscious  deliberation 

Study  how  emotion  affects  (cognitive)  task 
representation 

-  Implicit  vs.  explicit 


Defence  R&D  Canada  -  Toronto  •  R&  D  pom'  la  defense  Canada  -  Toronto 
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A3.5  BEHAVIOUR  GENERATION:  VARIABILITY  AND  CHOICE 


Behaviour  Generation 

Variability  and  Choice 
Joe  Armstrong 


HBR  Requirement 


-  are  more  complex  (include  relevant  perceptual, 
cognitive,  social,  personality  or  affective 
components), 

-  are  more  flexible  and  plausible, 

-  are  capable  of  extension  or  learning  to  support 
unscripted  behaviour. 
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Discussion 

•  Existing  analysis  techniques 

-  Development  &  Acquisition  Support 

-  Concept  Exploration 

•  Integration  within  models 

-  Behavioural  alternatives 

-  Behavioural  drivers 

-  Link  between  source  of  variability  and  cognitive  activity 

•  Characterization  of  behaviour 

-  Individual  vs.  group 

-  Relationship  to  error 

-  Military  vs.  Civilian 


Jgs  |nato 
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dstl 

CSEM 

Combat  System  Engineering  Model 

HFM-143:  30-31  May  2007 

Carol  Cooper  Chapman 

Aim 

"...  better  understand  the  end-to-end  SA 
process  and  the  link  to  decision-making  in  the 
surface  platform  combat  system  including 
human  elements.  ” 
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Behavioural  Variability 

Where  does  behaviour  come  from? 

Data  driven  -  primarily  bottom-up: 

OODA  lOOp  (Boyd) 

Endsley’s  (iggs)  SA  model 
Experience  driven  -  primarily  top-down: 

CECA  lOOp  (Bryant  et  at,  2004) 

RPD  Model  (Klein,  1993) 

Decision  ladder  (Rasmussen,  1993) 

30-3 1  May  2007 

UdLL  ©Dsti  2007  Unclassified 

Dstl  is  part  of  the 

Ministry  of  Defence 

Top-down:  CECA  Loop 


Critique 


Information 

Gathering 


Adapt 


Compare 


Explore 


^‘Directed  Telescope” 


From  Bryant  etal.  (2004,  p.109) 


dstl 


30-31  May  2007 
©  Dstl  2007 


Unclassified 


Dstl  is  part  of  the 
Ministry  of  Defence 
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CECA-based  Model 


Which  comes  down  to  ... 


A3 -34 


RTO-MP-HFM-1 43 


APPENDIX  3  -  PRESENTATION  SLIDES 


References 


Bryant,  D.J.,  Lichacz,  Hollands,  J.G.  and  Baranski,  J.V. 

(2004)  'Modeling  Situation  Awareness  in  an  Organizational  Context: 
Military  Command  and  Control’,  in  Banbury,  S.  and  Tremblay,  S. 
(eds)  A  Cognitive  Approach  to  Situation  Awareness:  Theory  and 
Application,  Ashgate:  Aldershot,  UK. 

Endsley,  M.R.  (1995)  Toward  a  Theory  of  Situation  Awareness  in 
Dynamic  Systems’,  Human  Factors,  vol.  37(1),  pp. 32-64. 

Klein,  G.A.  (1993)  'A  Recognition-Primed  (RPD)  Model  of  Rapid 
Decision  Making’,  in  Klein,  G.A.,  Orasanu,  J.,  Calderwood,  R.  and 
Zsambok,  C.E.  (eds)  Decision  Making  in  Action:  Models  and 
Methods,  Ablex:  Norwood,  NJ. 

Rasmussen,  J.  (1993)  ‘Deciding  and  Doing:  Decision  Making  in 
Natural  Contexts,  in  Klein  et  al.  (eds),  op  cit. 


dstl 


Unclassified 


RTO-MP-HFM-1 43 


A3 -35 


APPENDIX  3  -  PRESENTATION  SLIDES 


Variability  in  behaviour 

Dr.  Emiel  Ubink 


Variability:  Al,  HF  &  HBR 

•  Al:  what  to  do  (action  selection) 

•  HF:  howto  do  it  (performance,  intensity,  ..) 

•  HBR:  ‘what’  &  ‘how’  are  both  important  and  often  not 
separable 

2  30  May  2007 
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Tasks  &  Stress 


•  What  &  How  both  depend  on  relative  importance  of  "stimuli” 

♦  Tasks  proactive  behaviour 

•  “Stress”  -»  reactive  behaviour 


3 


30  May  2007 


tm 


Capabilities 


What  &  How  also  depend  on  availability  of  resources/capabilities 


Task 


Behaviour 


Stress  & 
Strain 


v> 


Capabilities 


30  May  2007  TW# 
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ORGANIZATION 


A3.6  KEYNOTE  SPEECH  -  DAY  2 


A  Road  Map  for  Human  Behaviour  Modelling 


M.  Greenley 

Vice  President  Modeling  &  Simulation 
May  31,  2007 


CAE 


CAE 


OBJECTIVE 


►  To  share  one  industry  perspective  on  the  very  near  term 
requirement  for  human  behaviour  representation 
technology  to  become  a  more  mainstream  commercial 
technology  in  the  modeling  &  simulation  community. 


CAE  Inc.  Copyright. 
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My  Industry  Context  ■ 


World  leading  portfolio  of  simulation  based 
training  technologies. 


Global  Training  Services  footprint. 


Operational 
Flight 

Reconfigurable,  Trainers 
deployable 
trainers 


Desktop  trainers 
(CAE  Si  infinity) 


Part  task 
trainers 

Integrated 
Procedures 
Trainers 
(CAE  Si  infinity) 


My  Industry  Context  -  2 


Stand 

Alone. 


Distributed. 


Massive 

Distributed, 

L-V-C. 


Vehicle  interacting 
with  terrain. 


Vehicle  interacting 
with  a  few 
other  vehicles 
(EN  and  FR). 


Vehicle  interacting  with 
multiple  entities, 

EN  and  FR  and  civilian, 
in  complex  terrain, 
joint  operations,  civilian 
emergency  personnel, 
terrorist  entities,  etc. 

CAE  Inc.  Copyright. 
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My  Industry  Context  -  3 


Old 

Methods 


Analysis  & 
Prototypes 


.-4. 


Field 

Exercises 


Drawings  & 
Prototypes 


Paper  Maps 
&  Overlays 


Live 

Training  with 
Real  Vehicles 


Modeling  & 
Simulation 


Methods 


Civilian 

Emergency 

Management 


Live 

Training  with 
Real  Vehicles 


The  breadth  and  complexity  of  M&S  application  requires  intelligent  human  behaviour. 


CAE 


BEST-™  BEST 


0  engenuity 


& 


My  Industry  Context  -  4 


Visual  Modeling 


Model  Libraries 


Scenario  Builder 


High  End  IG’s 


After 

Action  Review 
&  Analysis 


Integrated  Suite  of  Modelling  &  Simulation  Tools 


CAE  Inc.  Copyright. 
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Problem  Space 


Robotics 


Crowd 


CAE  Inc.  Copyright. 


Human  Behaviour  Modelling  -  Products 


►  Current  Market  Areas 

■  Human  Performance  Prediction 

■  Training  Systems 

■  Intelligent  Agents 

►  Estimated  Global  Market 

■  Difficult  to  Assess 

■  Government  R&D  Basis 

■  Few  mainstream  applications 


CAE  Inc.  Copyright. 
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ORGANIZATION 


Artificial  Intelligence 


Current  Market  Areas 

■  Dynamic/Intelligent  Systems 

■  Knowledge  Management/Decision  Aids 

■  Data  Mining 

■  Interactive  Media 

■  Economic  Forecasting 

■  Environmental  Prediction  &  Analysis 

■  Expert  Systems 

■  Automated/Autonomous  Systems 


►  Estimated  Global  Market 

■  -900$  Million  USD  (US  Department  of  Commerce,  1993) 
-  -11.9$  Billion  USD  (BCC,  2002) 

■  +21$  Billion  USD  (BCC,  2007) 

CAE  Inc.  Copyright. 


CAE 


►  Current  Market  Areas 


Robotics 


■  Industrial  Manufacturing  (Automotive,  Aerospace,  Marine) 

■  Autonomous  Systems 

•  Aerospace  and  Ground  Vehicles 

■  Medical  Systems 

■  Nanotechnology 

►  Estimated  Global  Market 

■  +16$  Billion  USD  (BCC,  2007) 


•  +24.3$  Billion  USD  by  2007 


CAE  Inc.  Copyright. 


A3 -42 


RTO-MP-HFM-1 43 


NATO 

OTAN 


APPENDIX  3  -  PRESENTATION  SLIDES 


CAE 


►  Current  Market  Areas 


CGFs 


■  Military  Simulation 

■  Emergency  Response 

■  Civilian  Response 


mixed  between  a  large  services 
market  around  the  application  of 
Government  Off  the  Shelf  (GOTS)  technologies 
and  a  smaller  COTS  technology  demand. 


■  Infrastructure  Analysis  (BP) 

►  Estimated  Global  Market 

■  Up  to  $1 B  of  global  activity 


CAE  Inc.  Copyright. 


CAE 


Crowd  Modelling 


►  Current  Market 

■  Simulation  &  Training 

■  Games  &  Entertainment 

■  Autonomous  System  Control 

♦  UAVs/UGVs 

♦  Nanotechonology 

■  Command  &  Control 

►  Estimated  Global  Market 

■  -3.5$  Million  USD  in  G&E 

■  Growing  new  markets  in  defense,  homeland  defense,  urban 
planning,  architecture,  among  others. 

CAE  Inc.  Copyright. 
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ORGANIZATION 


Problem  Space 


►  History  of  Development 

■  Requirements  evolved  from  military  simulations 

•  CGFs 

•  Performance  Prediction 

•  Acquisition  Support 

•  Training  Applications 

■  Theory  evolved  from  academia 

•  Cognitive  Science/Psychology 

•  Computer  Science 

■  Implementation  dependant  on  Industry 

•  Product  Development 

•  Software  Integration 

CAE  Inc.  Copyright. 


Problem  Space 


►  Current  Approach 

■  Fragmented 

■  Minimal  Cohesion 

■  Minimal  Coordination 

■  Difficult  to  Foresee  Future  Growth 

■  Reactive  -  Chasing  Opportunities 

■  Requires  Evolution 
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Problem  Space 


►  Fragmented  Industry 

■  Low  barriers  to  entry 

♦  Many  specialized  firms 

♦  Similar  to  fledgling  Al  industry  in  1980s 

■  Highly  Specialized  Market 

♦  Niche  oriented 

♦  Independent  evolution  of  technology 

♦  Multiple  toolsets  for  similar  requirements 

♦  Degendent  on  Services  for  implementation 

•  Lack  of  Standardization 

♦  Awareness  of  requirement 

♦  Problems  for  integration 


CAE  Inc.  Copyright. 
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ORGANIZATION 


►  Application  level 

■  Scaleable  Infrastructure 

■  Modular  applications 

■  Sustainable  long-term 
architectures 

■  Standards  definition  & 
development 


Scaleability.  Re-use. 


Benefits  of  Integration 


Aggregate 


CAE 


Benefits  of  Integration 


►  Organizational 

■  Growth  of  advanced  techniques 
from  academia 

•  R&D  Guidance  from 
Industry/Government 

■  Definition  of  requirements  in 
conjunction  with  Government 

°  Matched  with  Industry  Capabilities 

•  Supported  via  Academic  research 


■  Integration  of  techniques  and 
requirements  via  industry  =  Products 

■  Development  of  VV&A  Processes 

•  Theoretical,  Predictive 

•  Accreditation  based  on  user  application. 


I 


CAE  Inc.  Copyright. 


A3 -46 


RTO-MP-HFM-1 43 


APPENDIX  3  -  PRESENTATION  SLIDES 


RTO-MP-HFM-1 43 


A3 -47 


APPENDIX  3  -  PRESENTATION  SLIDES 


A3.7  MILITARILY  RELEVANT  MENTAL  OUTPUT  MEASURES 


Militarily  relevant  mental 
output  measures 


NATO  Specialist  Meeting 
Toronto  May  31  2007 
Andy  Belyavin,  QinetiQ 


Militarily  relevant  mental  output 
measures 

•  Objective  of  developing  a  model  is  to 
predict  overall  system  performance  at 
some  level 

•  For  low  level  systems  involving  human 
crew  two  important  constructs  are 
described  and  measured  in  live  studies: 
-Workload  and  Situational  Awareness 

•  Discuss  these  issues  in  the  following 
slides 
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Workload 


•  A  measure  of  resource  demand  on  the 
members  of  the  crew 

•  Important  element  is  time 

•  Definition  of  workload  involves  the  idea  of 
rate  of  resource  consumption  per  unit  time 

•  Less  than  1 00%  -  fine  >  1 00%  problems 


Modelling  workload 


•  Many  models  of  workload  have  been  developed 

•  POP,  IP/PCT,  POPIP,  W/Index,  VACP 

•  POP  predicts  changes  in  performance  does  not 
address  scheduling 

•  POPIP,  IP/PCT  models  scheduling 

•  Is  a  model  of  workload  useful  if  it  does  not 
predict  performance/behaviour  effects? 

•  If  we  can  predict  performance  do  we  need  to 
predict  a  measure  of  workload? 
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ORGANIZATION 


Situational  Awareness 

Best  definitions  of  SA  articulate  the  basic 
idea  that  people  need  understanding  to 
make  good  decisions 

Understanding  can  be  defined  as  having  a 
mental  model  of  how  the  “world”  will 
evolve  that  is  consistent  with  reality 

Access  to  and  acquisition  of  relevant 
information  is  part  of  the  problem  but  is  not 
sufficient 


NATO 


OTAN 


ORGANIZATION 


Metrics? 

We  can  model  the  possession  of  a  good 
mental  model 

In  principle  could  define  metrics  for  a 
model  that  we  cannot  define  for  live 
experimental  subjects 

Is  that  useful? 

If  behaviour  is  good  and  performance  is 
good  do  we  need  to  define  the  abstraction 
comprising  SA? 


NATO 


OTAN 
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Basis  for  discussion 

Is  Workload  a  well-defined  construct? 
Do  we  need  to  model  it  and  if-so  how? 

Do  we  need  to  validate  workload 
predictions  from  models? 

Is  SA  a  well-defined  construct? 

Can  we  validate  measures  of  SA? 

Is  that  a  useful  activity? 

Are  there  other  measures? 


I  NATO 
OTAN 
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Workload  Concepts 


Operator  Workload 

Non-uniform  concept  (Hart  &  Staveland,  1988;  Xie  &  Salvendy,  2000) 
Hypothetical  Construct  vs  Intervening  Variable  (Gopher  &  Donchin,  1986) 
Meta-cognitive 

Multi-Dimensional 

Task  demands  vs  resources  (Wickens,  2002;  Young  &  Stanton,  2002) 
Dynamic  &  Static  Attributes  (North  &  Riley,  1989;  Hendy  &  Farrell,  1997) 
Non-attentional  parameters  (Kahneman,  1973;  Meshtaki,  1988) 

•  Task  environment 

•  Moderating  variables  (e.g.  arousal,  motivation,  emotion) 
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Workload  Measurement  Techniques 


►  Categories  of  Workload 
Measurement 

■  Workload  Assessment 

•  Measurement  of  dynamic  operator 
workload  in  complex  system 

•  Can  be  applied  to 

-  Real,  Applied  Situations 

-  Simulated  Environments 

■  Workload  Prediction 

•  Development  of  computational 
models  of  workload 

•  Used  to  Predict  performance  a  priori 

-  Models  of  Human 
Behaviour/Cognition 

-  Synthesized  Environments 


Domains  /  Scenarios  /  Tasks 


Algorithmic/Predictive  Measures 


►  Applications 

■  Diagnostic  workload  predictions 

■  Implemented  within  Computational  Simulations 

•  Mathematical/Algorithmic 

•  Task  Analysis 

•  Computer  Simulation 

►  Major  Categories 

■  Discrete  Event  Simulations  (Task  Network  Modelling) 

■  Cognitive  Architectures 


CAE  Inc.  Copyright 
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ORGANIZATION 


Task  Network  Modelling 


►  General  Assumptions 

■  Human  Behaviour  Modelled  as  Interrelated  Tasks 

■  Performance  Values  Assigned  by  Developer 

♦  Time/Accuracy 

♦  Task  Demands 

■  Sequences  managed  by  a  discrete  event  simulator 

♦  IPME 

♦  IMPRINT 

♦  SAINT/MicroSAINT 


CAE  Inc.  Copyright 


General  Model 


Process  Request, 
Develop  Plan 


I 


Data  Processing,  Exploitation 

t 


Fusion 


I 


Dissemination 


VIS -Visual 
AUD  -  Auditory 
COG  -  Cognitive 
MOT  -  Psychomotor 
Within 
Interference 
Workload 
Num  Tasks 


Attentional  Demand 
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CAE 


Scheduler/Performance  Algorithms 


►  Qualitatively  Different  than  VACP  or  W/INDEX 

■  Measure  impact  of  task  demands  on  task  performance 

■  Simulates  scheduling  of  tasks  based  on  demand 

►  Two  Major  Theories 

■  IP/PCT 

■  POP  &  POPIP 


CAE  Inc.  Copyright 


Single  Task 
Performance 


Processing 

intervals 


Dual  Task 
Performance 


Task  I 

Task/ 


Task  / 

Task  j 


Response  Time 


Time-multiplexing  in  concurrent  task  processing  (Hendy,  2003) 
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ORGANIZATION 


Cognitive  Architectures 


►  General  Assumptions 

■  Theories  of  cognition  (Keiras  &  Meyer,  1997) 

■  Not  workload  specific 

•  By-product  of  overall  performance 

•  Intervening  variable/hypothetical  construct 

►  Major  Players 

■  ACT-R  (Anderson,  1993) 

■  SOAR  (Laird,  Newell,  and  Rosenbloom,  1987) 

■  COGNET  (Zachary,  Ryder,  Ross  &  Weiland,  1992) 

■  EPIC  (Keiras  &  Meyer,  1997) 


Cognitive  Architectures 


►  Relation  to  Workload 

■  Functional/Task  distinction 

♦  Task  behaviour  related  to  knowledge 

-  Procedural/declarative 

♦  Functional  components  of  cognition 

-  Perception,  Audition,  Psychomotor  Functions 

■  Task  performance 

♦  Inferred  from  operation  of  architecture 


CAE  Inc.  Copyright 
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Validation  Overview 


CAE 


►  Contrast  of  Verification  &  Validation 


Is  the  model  a  reflection  of  the  task  environment? 
(Verification) 

Vs 

Does  the  model  predict  reality? 
(Validation) 


CAE  Inc.  Copyright 


Concepts  of  Validation 


►  Criterion 

■  Predictive  and  Concurrent  (Cronbach  &  Meehl,  1955) 

■  Task  Performance  vs  Subjective  Estimates 

►  Content 

■  Measuring  internal  constructs  (e.g.  task  performance/resource 
demand  is  a  measure  of  workload) 

►  Construct 

■  Computational  Model  =  Workload  Construct 
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ORGANIZATION 


Model  Validation 


Construct  Validity 


Predictive  Validity 


Validation  Framework 


Values  Consistent  with  Theory 
„  Inter-Rater 


Rating  Allocation 


Algorithms 


'State  Predictions 


‘Intra-Rater 


Subjective 

Performance 


Performance  Predictions 


Primary 

Secondary 
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A3.8  THE  CONCEPT  OF  MODERATORS 


Concept  of  moderators 


NATO  Specialist  Meeting 
Toronto  May  31  2007 
Andy  Belyavin,  QinetiQ 


I NATO 
^f7  OTAN 


ORGANIZATION 


Concept  of  moderators 

Human  performance  differs  from  the  performance  of 
physical  components  in  a  system  in  that  it  changes  with 
well-known  effects  such  as  fatigue  or  other 
environmental  stress. 


Different  individuals  perform  and  behave  differently  in 
the  same  context  so  the  full  spectrum  of  human 
variability  involves  both  inter-  and  intra-individual 
variability. 

The  drivers  of  these  differences  are  termed  moderators 


|  NATO 
OTAN 


ORGANIZATION 
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ORGANIZATION 


Classification  of  moderators 


Moderators  can  be 
allocated  to  three 
groups: 

-  external  moderators 
(stressors) 

-  internal  moderators 
(personal  attributes) 

-  collective  moderators 


Individual  stressors 


I  NATO 
OTAN 


ORGANIZATION 


Key  external  moderators 


•  Environmental  impact: 

-  Sleep  loss  fatigue/circadian  effects  and  time 
on  task 

-  Physical  fatigue 

-  Thermal  effects  (thermal  strain, 
dehydration,  discomfort) 

-  Effects  of  visual  environment 

-  Fear,  anxiety,  morale 

-  Task  demand — workload 

Sample  development  since  1995 

•  Development  of  SAFE  by  DERA/QinetiQ  to 
model  the  effects  of  fatigue,  circadian  rhythm 
and  time-on-task  implemented  in  IPME 

•  Development  of  thermal  models  that  can  be 
coupled  to  task  network  models. 

•  BAE  SYSTEMS  ORACLE  vision  model  coupled 
to  IPME 

•  DERA/QinetiQ  developed  the  Prediction  of 
Operator  Performance  (POP)  Workload  model 
-validated  in  experiments 


1 

NATO 

OTAN 

ORGANIZATION 
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Detailed  model  of  sleep  loss 


■JNATO 
^  OTAN 


ORGANIZATION 


Key  internal  moderators 


•  Personal  characteristics 

-  Training 

-  Experience 

-  Personality  -  including  coping  style  and 
culture 

-  General  intelligence 

Sample  Development  since  1995 

•  Development  of  the  Performance  Shaping 
Function  for  Experience  by  QinetiQ 

•  Development  of  Performance  Shaping 
Function  for  the  effects  of  training  based  on 
standard  results  by  Micro-Analysis  and 
Design 

•  Representation  of  effect  of  skills  in  IMPRINT 
to  enable  allocation  of  roles 


ORGANIZATION 
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Key  collective  moderators 


Collective  characteristics 

-  Training 

-  Experience 

-  Ad-hoc  team 

-  Cohesion 

-  Leadership 

-  Culture/Organisation 


Time  in  Army  against  LER 


71  71.5  72  72.5  73 

Time  in  Army  (months) 


73.5  74 


^NATO 
OTAN 


ORGANIZATION 


Modelling  moderator  effects 

Modelling  the  effects  of  external  moderators  is 
best  developed  area 

Many  stressors  have  been  the  subject  of  detailed 
research  for  more  than  60  years 

Can  build  good  physiological  models  and  model 
effects  of  stressors  on  state;  state  to 
performance  and  behaviour  not  as  good 

Other  areas  not  as  advanced 


jJBnato 
WP  OTAN 


ORGANIZATION 
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Problems 


•  Defining  a  minimal  set  of  states  that  affect 
performance  and  behaviour  is  not 
complete 

•  The  impact  of  even  those  states  that  have 
been  identified  is  not  well-defined 

•  Behavioural  effects  are  least  developed 

•  A  data-limited  area 


B 


Questions 


•  For  what  applications  do  we  need  to  solve 
the  moderator  problem? 

•  Can  we  ignore  moderators  across  a  range 
of  problems? 

•  How  do  we  develop  valid  models  if  we 
need  to? 

•  How  do  we  deal  with  potential  moderator 
interactions? 
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r 

A  word  or  two  about  moderators 

or  a  bird  in  the  hand... 

Dr.  Laurel  Allender 


The  idea 

■  Take  published  data 

■  Derive  general  degradation  factors 

■  Associated  with  different  task  types 

■  Take  modeled  tasks 

■  Denote  as  task  type  X  (up  to  3  task  types 
with  weights  =  1) 

■  Apply  degradation  factor  to  modeled  task 

■  Run  model  and  compare  with  non-degraded 
baseline 
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[pTS 

■  Personnel  characteristics 

o  Data  collected  on  9000  Soldiers,  updated  w/ 
longitudinal  data 

■  Training 

o  Recently  updated 

■  Stressors 

o  Heat  +  humidity 
o  Cold  +  wind  chill 
o  Noise 

o  Sleeplessness 
o  Protective  clothing 


Task  taxonomy 

■  Fine  motor  discrete 

■  Fine  motor  continuous 

■  Gross  motor  light 

■  Gross  motor  heavy 

■  Visual 

■  Auditory 

■  Cognitive  -  decision  making. . . 
o  (a  la  Berliner,  see  Fleishman) 
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The  general  formula  (notional  data) 

■  Performance  moderator  effects  on 

tasks  as  a  function  of  task  type 

o  e.g.,  hours  since  last  sleep  20  hours  = 
performance  decrease  of  5%  for 
cognitive  tasks 

o  derived  from  published  data 

o  %  decrease  applied  to  performance 
estimates  attached  to  cognitive  tasks  in 
network  model 


An  existinq  modelinq  tool 

IMPRINT 

Perl  d  nun  it  hw».»  larttuM*  Knl 

■  with  numerous  applications  to  Army 
systems 
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A3.9  COMPLEXITY,  HIERARCHY,  MODULARITY  AND  VALIDITY  IN  HER 
ARCHITECTURES 


Institute 

of 

Cognitive 

Science 

On  the  Validity  of  Cognitive  Models 

0  Carleton 

UNIVERSITY 

Canada's  Capital  University  Robeft  L.  West 

Institute 

of 

Cognitive 

Science 

types  of  validity 

■  Face  validity 

-  Model  seems  realistic  to  people  who  have  done  the  task 

■  Model  validity 

-  The  model  fits  the  task  data  (without  extensive  parameter 
fitting) 

-  Data  sets 

0  Carleton 

IT  UNIVERSITY 

Canada's  Capital  University 

-  Better  ways  of  fitting 

■  Predictive  validity 

-  The  model  can  fit  other  data  from  the  same  task  (without 
major  changes) 

-  The  model  makes  a  novel  prediction  that  can  be  tested 

RTO-MP-HFM-1 43 


A3 -67 


APPENDIX  3  -  PRESENTATION  SLIDES 


ORGANIZATION 


Institute 

types  of  validity 

of 

Cognitive 

■  Architecture  validity 

Science 

-  Model  uses  a  well  researched  architecture  (with  no  hidden 
tricks) 

—  Good  architecture 

■  Many  studies  showing  that  it  accurately  predicts  human 
behavior  across  a  wide  variety  of  tasks 

1  Carleton 

■  Established  parameter  values 

UNIVERSITY 

Canada’s  Capital  University 

■  Neural  correlates 

-  Offloads  validity  onto  universities 

■  If  the  architecture  is  open  source 

-  E.g.,  SOAR,  EPIC,  LIBRA,  ACT-R 

3 

Institute 

of 

Cognitive 

Science 


types  of  validity 

■  Agent  validity 

-  Model  is  consistent  with  other  models  of  similar  tasks 

-  Goal  should  be  to  develop  agents  that  have  general  skills 
■  e.g.,  driving 

■  Comparative  validity 

-  Are  other  models  of  the  same  thing  similar  or  different? 
Carleton  -  Is  there  convergence? 

-  If  not  is  there  some  way  to  compare  competing  models 

■  Publishing 

-  Models  need  to  be  accessible 


Canada's  Capital  University 


A3 -68 


RTO-MP-HFM-1 43 


APPENDIX  3  -  PRESENTATION  SLIDES 


Institute 

of 

Cognitive 

Science 

Thank  You 

Questions?? 

0  Carleton 

UNIVERSITY 

Canada's  Capital  University 
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Modular  HBR 


Dr.  Emiel  Ubink 


The  near  in  near  decomposability 


TH« 


TNO  I  Knowledge  for  business 


Near  decomposable  human  modelling 


Human  model 


Physiology 


Biomechanics 


Metabolism 


Equipment 


Clothing 


Weaponry 


Cognition 


Memory 


Perception 


Behaviour 


Some 

Behavioural 

Architecture 


30  May  2007  TW# 
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Near  decomposable  behaviour  modelling 


Nearly  decomposable  model 


threat 


fatigue 


Black 

board 


Black 

board 


Black 

board 


Rule 

base 

Rule 

base 

Rule 

base 

3 


30  May  2007 


TH9 


The  near  in  nearly  decomposable 


4  30  May  2007  TO# 
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L  )  cwimr.^)  Canmlt  A  1 

Uicowlder 

(O  Latent  (£)  Activated  1 

CD  Retrieved 

Comparative  Analysis  of 
Frameworks  for  Knowledge- 
Intensive  Intelligent  Agents 

Randolph  M.  Jones 

Robert  E.  Wray 

30-31  May  2007 

NATO  Specialists  Meeting  HFM-143/RSM  On 

Human  Behaviour  Representation  in  Constructive  Modeling 
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Motivations 

■  Software  engineering  principles  have  been  applied 
mostly  to  "light"  agent  architectures 

■  Cognitive  and  agent  architectures  share  more  with 
each  other  than  commonly  recognized 

•  How  can  we  evaluate  until  we  can  compare? 

•  We  need  a  common  language  for  intermediate 
components 

■  Better  reuse  would  benefit  applications 
development  as  well  as  science 

•  Easier  to  build  new  models  and  new  architectures 

•  Easier  to  integrate  models  into  applications 

•  Easier  to  communicate  success  to  broader  communities 
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Knowledge-Rich 
Intelligent  Agents 

■  Significant  encodings 
of  knowledge 

•  Contrast  with 
"Light"  agents:  limited, 
restricted  capabilities 

•  Intended  to  perform 
the  role  of  one  or  more 
humans 

■  Integrated  with 
non-trivial  environments 

•  Dynamic,  unpredictable,  inaccessible,  continuous 

•  Often  require  real-time  response 

■  Situation  interpretation/representation  a  essential  part  of  decision 
making 

■  Long-lived  ("enduring")  in  many  dimensions 

•  Execution  time 

•  Software  life  cycle 

©  2004  Soar  Technology,  Inc.  ♦  May  30-31 , 2007  ♦  NATO  HER  Specialists  Meeting  ♦  Slide  3  S03f  TBChflOlOOV 


Goals  for  Architectures 
for  Intelligent  Behavior 

■  Comprehensive 

•  Explicit  support  of  common  architectural  processes  and 
representations 

■  Scalable 

•  Engineered  for  efficiency  and  composability 

•  Designed  around  abstract  component  interfaces 

■  Stable 

•  Well-defined  abstract  component  types 

•  Well-engineered  implementation 

■  Easy  to  use 

•  Extensible 

•  Integrated  design  with  high-level  formal  abstractions 
(languages,  libraries,  tools) 
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Assessing  the  State  of 
the  Art 

■  No  (existing)  architecture  supports  all  basic 
representations  &  processes  needed  for 
most  applications 

■  Large  gap  between  notional  architecture 
components  and  software  implementation 

•  Few  intermediate,  reusable  structures  and 
components 

■  Not  engineered  for  real-world  applications 

•  Robustness,  ease-of-use,  scalability,  stability 

Incomplete  Support  for 
Basic  Representations 

■  Examples  of  missing  representations  & 
functionality 

•  Initial  analysis:  BDI,  GOMS,  Soar 

•  Analytical  framework  inspired  in  part  by  BDI  (Beliefs, 
Desires,  Intentions,  Commitment,  Reconsideration) 

■  Consequences: 

•  Labor  intensive,  ad  hoc  design  &  development 

♦  Where  the  architectures  lack  representational  or  processing 
power,  users  must  "program"  solutions  to  gaps 

♦  Exacerbates  "tangling"  between  domain  knowledge 
representations  &  gap-driven  solutions 

•  Minimal  knowledge  reuse 

♦  Ad  hoc  development  not  applicable  in  new  task 
environments 
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Comparative  Framework 

■  Representational  elements 

•  Inputs,  Justified  Beliefs,  Assumptions,  Desires,  Active 

Goals,  Plans,  Actions,  Outputs 

•  Superset  of  GOMS,  Soar,  BDI  representations;  not  an 
exhaustive  list 

■  Design  dimensions 

•  Representation  formalism 

♦  How  is  each  type  of  element  represented? 

•  Commitment  strategy 

♦  Under  what  conditions  does  each  type  of  element  get 
selected/activated/instantiated? 

•  Reconsideration  strategy 

♦  Under  what  conditions  does  each  type  of  element  get 
removed/deactivated/released? 

©  2004  Soar  Technology,  Inc.  ♦  May  30-31 , 2007  ♦  NATO  HER  Specialists  Meeting  ♦  Slide  7  A  S03f  TBChnOlOQY 

Abstract  Knowledge  Cycle 
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Reusable  Abstract 
Components 

■  Architectures  share  only  minimal  high-level 
commitments/components 

•  Physical  symbol  systems,  problems  space  hypotheses 

•  Long-term,  short-term  memory 

•  Notable  exceptions 

♦  Prodigy:  Focus  on  interoperation  between  capability- 
level  modules  (e.g.,  EBL  and  planning) 

♦  PRISM:  Functional,  parameterizable  abstractions  of 
common  architectural  components 

■  Consequences: 

•  Little  transfer  from  one  architecture  to  another 

•  Hard  to  understand  architecture,  changes  to  architecture 
(even  among  user  communities) 

•  Difficult  to  investigate  integration  of  new  or  alternatively 
designed  architectural  components 
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Proposed  Conceptual 
Components  &  Relationships 


Commit:  Persistent  assertion 
Assum  ption 


Commit:  Truth  maintenance 
Entail  me  nt 


Recon:  Persistent  assertion 


Recon:  Truth  maintenance 


Commit:  Activation  threshold 
Activated 
Assertion 

Recon:  Activation  threshold 


Commit:  Persistent  assertion 
Context-Sensitive 
Assumption 
Recon:  Persistent  assertion 
Recon:  Truth  Maintenance 


Commit:  Persistent  assertion 
Te  mporal 
Assumption 
Recon:  Decay  function 
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Summary 


■  Intelligent  agent  architectures  are  complex  software  systems  and 
require: 

•  Comprehensive  support  for  necessary  knowledge 
representations 

•  Composable,  well-defined  software  components 

•  Design  that  addresses  understandability  and  usability 
challenges  from  the  outset 

■  Long-term  strategy 

•  An  abstract  machine  based  on  common  functional  components 
(e.g.,  commitment,  beliefs) 

•  Interoperable  object  libraries  as  instantiations  of  abstract 
components  (e.g.,  preference- mediated  deliberation,  FOPC 
sentences) 

•  Formal  framework  to  provide  a  bridge  between  science  and 
implementation 

•  Ability  to  compose  components  quickly  into  the  "best" 
architecture  for  a  given  task 
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Extra  Slides 
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Unifying  View  of  Memory 
Operations 

CCRU  formalism  for  all  architectural  components 
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Review  of  Selected  Agent 
Frameworks 

■  BDI 

•  Inspired  by  formal  logic/philosophy 

•  Formal  sense  of  rationality 

•  Focus  on  logical  consistency  between  beliefs  and  goals 

■  GOMS 

•  Inspired  by  psychology 

•  Explicit  hierarchical  task  decomposition 

•  Explicit  pairing  of  goals  with  plans 

■  Soar 

•  Inspired  by  functionality  and  philosophy 

•  Problem-space  hypothesis 

•  Physical  symbol  systems  hypothesis 

•  Focus  on  minimal  but  sufficient  set  of  principles 
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Explicitly  supported 

Partially  supported 


Base-level 

Representation 

Representation 

Language 

Mechanism  of 
Commitment 

Mechanism  of 
Reconsideration 

Inputs 

BDI 

Input  Language 

GOMS 

Input  Language 

Soar 

Working  memory 

Justified 

Beliefs 

BDI 

Beliefs 

Logical  Inference 

Belief  revision 

GOMS 

Working  memory 

Match -based 
assertion 

Soar 

Working  memory 

Match -based 
assertion 

Reason  maintenance 

Assumptions 

BDI 

Beliefs 

Plan  language 

Plan  language 

GOMS 

Working  memory 

Operators 

Operators 

Soar 

Working  memory 

Deliberation/ 

operators 

Operators 

Desires 

BDI 

Desires 

Logic 

Logic 

GOMS 

Soar 

Proposed  operators 

Preferences 

Preferences 
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ORGANIZATION 


Explicitly  supported 

Partially  supported 


Base-level 

Representation 

Representation 

Language 

Mechanism  of 
Commitment 

Mechanism  of 
Reconsideration 

Active  Goals 

BDI 

Intentions 

Deliberation 

Soundness/Decision 

Theory 

GOMS 

Goals 

Operators 

Soar 

Beliefs/lm  passes 

Deliberation 

Reason  maintenance 

Plans 

BDI 

Plans 

Plan  selections 

Soundness 

GOMS 

Methods 

Selection 

Soar 

Interleaving 

Actions 

BDI 

Plan  language 

Atomic  actions 

GOMS 

Operators 

Operators 

Soar 

Primitive  operators 

Deliberation 

Reason  maintenance 

Outputs 

BDI 

GOMS 

Plan  language 

Plan  language 

Soar 

Working  memory 

Conditional 

operators 

(decoding) 
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Reference: 

■  Jones  and  Wray  (2006)  Comparative 
analysis  of  frameworks  for  knowledge- 
intensive  agents.  AI  Magazine 

©  2004  Soar  Technology,  Inc.  ♦  May  30-31, 2007  ♦  NATO  HER  Specialists  Meeting  ♦  Slide  18  SoarTBChllOlOOY 

A3 -80 


RTO-MP-HFM-1 43 


APPENDIX  3  -  PRESENTATION  SLIDES 


A3.10  FROM  INDIVIDUAL  TO  GROUP  BEHAVIOUR 


Topic  7 

From  individual  to  group  behaviour 

Brad  Cain 


| NATO 
^  OTAN 


ORGANIZATION 


Military  simulation: 
The  early  days 


Lanchester  models 
dx/dt  +  ky  =  0 
dy/dt  +  mx  =  0 

Attrition  models  of  Force-on-Force 
Little  in  the  way  of  “Human  Factors” 

Human  elements  are  aggregated  in  the  various  coefficients 


|  NATO 
OTAN 


ORGANIZATION 
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1 


Small  teams 


•  Representation 

-  Single  “team”  entity 

-  Collection  of  cooperating  individuals 

•  Individual  HF  aspects  may  still  be  relevant 

-  fatigue,  thermal,  etc. 

•  Each  approach  has  pros/cons 

-  entity:  conceptually  simpler  but  harder  to  validate 

-  individuals:  shared  understanding  &  comms  but  more 
detailed  models  required  to  represent  interactions 


Additional  team  attributes 


•  Additional  social  elements 

-communication,  feedback 

-  mutual  support,  monitoring,  task  sharing 

-  morale,  commitment 
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Large  teams  and  organizations 


•  Social  elements  apply  to  organizations 

-Human  Factors  modelling  requirements 

•  goals,  procedures,  errors,  social  elements 

•  business  process  models 

•  elaborate  Lanchester  models? 

-Are  there  HBR  requirements  in  organizational 
models? 


•  perception,  reasoning,  performance 

•  fatigue,  thermal 


r 

1 

NATO 

P 

1 

OTAN 

ORGANIZATION 


Team  models  (and  organizations?) 


Can  we  define  those  essential  attributes  of  teams  that  require  formal 
models  that  would  make  team  entity  modelling  viable? 

-  Are  validated  formal  models  available? 

Can  we  model  teams  of  individuals  at  sufficient  resolution  and 
validate  them  for  use  in  military  simulations? 

-  What  can  make  this  process  affordable? 

Can  we  define  the  pros  and  cons  of  each  approach  sufficiently  so 
that  the  military  M&S  community  can  make  an  informed  decision 
about  which  is  the  more  appropriate  in  a  given  context? 

How  is  an  organization  representation  different  from  a  small  team? 

-  Do  we  need  to  represent  explicit  EBO  behaviours  or  is  it  sufficient  to 
represent  their  “effects”? 


NATO 


OTAN 


ORGANIZATION 
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Institute 

of 

Cognitive 

Science 

Modelling  in  Sociotechnical  Systems 

0  Carleton 

UNIVERSITY 

Canada’s  Capital  University  Robeit  L.  West 

Gabriella  Nagy 

Institute 

of 

Sociotechnical  Systems 

Cognitive 

■  Individuals,  groups,  technology. 

Science 

administration,  command  and  control 

■  Dynamically  interacting  with  each  other  and 

with  the  environment 

0  Carleton 

IT  UNIVERSITY 

■  Includes  the  A  space  and  the  B  space 

Canada’s  Capital  University 

-  If  you  are  in  B  can  your  model  work  in  A 

-  Can  A  work  if  B  is  wrong 

-  Can  you  include  a  link  to  A  in  B 

2 
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Institute 

nf 

GOMS 

UI 

Cognitive 

■  Card,  Moran,  &  Newell 

Science 

■  Goals,  operators,  methods,  selection  rules 

■  Very  representative  of  the  type  of  model  generally 
used  for  military  simulations 

■  GOMS  models  applied  to  multi-agent  sociotechnical 

0  Carleton 

UNIVERSITY 

systems 

Canada’s  Capital  University 

—  West  &  Yeun 

-  Kieras  &  Santoro 

■  GOMS  does  not  work  well ! ! ! ! 

3 

Institute 

of 

Cognitive 

Science 

Hierarchical  goal  based  systems 

■  These  do  not  work  in  sociotechnical  system  because 

of: 

-  Frequent  interruptions 

-  Frequent  opportunistic  task  switching 

■  Therefore,  models  of  individual  behavior  built  in  this 

0  Carleton 

IT  UNIVERSITY 

Canada’s  Capital  University 

way  will  not  reflect  real  behavior  in  sociotechnical 
systems 

■  Can  this  be  fixed????? 

4 
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Institute 

of 

Cognitive 

Science 

SGOMS 

■  Sociotechnical  GOMS 

-Augmented  GOMS  to  cope  with 

■  Interruptions 

■  Task  switching 

-Not  GOMS  specific 

0  Carleton 

UNIVERSITY 

Canada’s  Capital  University 

■  Will  work  with  any  hierarchical  goal  based  system 

5 

Institute 

of 

Cognitive 

Science 


??  Carleton 

IT  UNIVERSITY 
Canada's  Capital  University 


SGOMS 

■  Unit  tasks  -  cognitive  level 

-  GOMS  -  prevents  overload  and  downtime 

-  SGOMS  -  units  of  work  that  are  not  meant  to  be  interrupted 

■  If  interrupted  they  must  be  abandoned  or  finished  before 
moving  on 

■  Planning  unit  -  social  unit 

-  Meaningful  units  of  work  composed  of  unit  tasks 

-  Used  for  planning  work  activities 

-  Used  to  manage  interruptions  and  task  switching 
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of 

Cognitive 

Science 

Planning  Units 

■  Planning 

-  Each  planning  unit  is  associated  with  a  set  of  constraints 

-  Plans  are  made  by  assigning  planning  units  to  groups  or 

individuals 

■  InteiTuptions 

—  Planning  units  can  be  interrupted 

0  Carleton 

UNIVERSITY 

-  Unit  task  is  completed  or  abandoned 

Canada’s  Capital  University 

-  New  planning  unit  is  selected  based  on  constraints 

-  The  new  planning  unit  may  have  already  been  worked  on 

7 

Institute 

of 

Cognitive 

Science 

Telcom  Study 
■  West  &  Nagy  (2007) 

-  Journal  of  Cognitive  Engineering  and  Decision  Making, 

Issue  2 

-  Tested  SGOMS  on  network  maintenance  teams  at  a  large 
Canadian  telecommunications  company 

0  Carleton 

IF  UNIVERSITY 

Canada’s  Capital  University 

8 
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GOMS  (West  &  Nagy,  2007) 


Institute 

of 

Cognitive 

Science 


S  Carleton 


UNIVERSITY 
Canada’s  Capital  University 


Label  Equipment 


Create  LAN  Diagram  ^77777 


Update  Database  for  Circuits  -  B 
Update  Database  for  Circuits  -  A  ^^777] 
Update  Database  for  Generics  -  B  ^^^^7^ 


Update  Database  for  Genencs  -  A  } ^^Eb77777\ 


Turn-Up  System  9999^777: 


Configure 


Obtain  IP  Address  ^^^9777? 
Lay  Cabling  -  D  P/777: 

Lay  Cabling  -  C  Rzz 
Lay  Cabling  -  B 


Lay  Cabling  -  A  gj 


Install  mb 
Get  the  Equipment  -  B  P/// 
Get  the  Equipment  -  A 


Understand  the  Project  -  E  ^^7-777-Am 
Understand  the  Project  -  A  ^//// 


H  Predicted  Steps  ■  Observed  Steps  Steps 
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SGOMS  (West  &  Nagy,  2007) 


Institute 

of 

Cognitive 

Science 


Carleton 

UNIVERSITY 
Canada's  Capital  University 


Label  Equipment 
Create  LAN  Diagram 
Update  Database  for  Circuits  -  B 
Update  Database  for  Circuits  -  A 
Update  Database  for  Generics  -  B 
Update  Database  for  Generics  -  A 
Turn- Up  System 
Configure 
Obtain  IP  Address 
Lay  Cabling  -  D 
Lay  Cabling  -  C 
Lay  Cabling  -  B 
Lay  Cabling  -  A 
Install 

Get  the  Equipment  -  B 
Get  the  Equipment  -  A 
Understand  the  Project  -  B 
Understand  the  Project  -  A 


EJ  Predicted  Steps  in  order  ■  Observed  Steps  in  order 
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Institute 

of 

Cognitive 

Science 

Conclusions 

■  Hierarchical,  goal  orientated  task  models  do  not 

predict  human  behavior  in  sociotechnical  system 

■  These  models  are  not  wrong,  just  incomplete 

-  They  seem  correct  to  workers  because  they  map  onto  how 
they  think  about  the  task 

0  Carleton 

UNIVERSITY 

Canada’s  Capital  University 

Institute 

of 

Cognitive 

Science 

Thank  You 

Questions?? 

0  Carleton 

IF  UNIVERSITY 

Canada's  Capital  University 
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ORGANIZATION 


DEFENCES 


DEFENSE 


Representation  of  Chilian  activity  for  the  Canadian  Army  synthetic 

environments 

Jerome  Levesque 

DRDC-CORA 

Land  Capability  Development  Operational  Research  Team 
CFB  Kingston,  Ontario 


Defence  Research  and  Recherche  et  developpement 
■  >  ■  Development  Canada  pour  la  defense  Canada 


Canada 


P^jJ)  Background 


\  J 


The  Directorate  of  Land  Synthetic  Environments 
(DLSE)  is  responsible  for  organizing  simulation- 
based  training  and  experimentation  for  the 
Canadian  Army. 


25  weeks  of  training  are  organized  every  year, 
primarily  at  the  battlegroup  level. 


3  major  experiments  are  funded  per  year, 
involving  tens  to  hundreds  of  participants,  plus 
about  ten  limited  objectives  experiments  (LOEs). 


Defence  R&D  Canada  -  CORA  •  R &  D  pour  la  defense  Canada  -  CARO 
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Constructive  simulation  softw  are  \ 

•  JCATS:  Interactive  land  combat  simulation  war  game 

-  Top-down  view 

-  Units  are  represented  by  icons 

-  Interactors  decide  their  actions  (with  lots  of  left/right  clicks) 

-  Main  differences  between  this  war  game  and  RTS: 


/'T'n 

Simulation  of  urban  environments 

•  Urban  environments  involve  civilian  activity. 

-  Circulation  of  persons 

-  Urban  traffic 

-  Formation  of  crowds,  disperse  to  dense,  passive  to  aggressive. 

NOTE:  crowd  dynamics  is  but  one  aspect  of  civilian  activity. . . 


Before  a  model  is  built  we  need  to: 

-  Lay  down  the  training/experimentation  objectives. 

-  A  software  platform  must  be  chosen. 
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V 


✓ 


Designing  a  civilian  activity  model  requires 
the  assessment  of  several  human  factors 

•  From  ail  experimentalist’s  perspective: 


Of  all  the  important  features  an  actual  population  exhibits, 
which  ones  can  reasonably  be  represented  by  an  agent- 
based  simulation? 

Which  simulation  outcomes  are  predictive  and  which  are 
not? 


From  a  trainer’s  perspective: 


How  can  we  teach  real-world  skills  using  a  2D 
representation? 

Which  features  should  the  2D  model  exhibit  to  maximize 
the  training  payoff? 


Defence  R&D  Canada  -  CORA  •  R&  D  pour  la  defense  Canada  -  CARO 


r^^y7  The  civilian  activity  modelling 
^  project  in  a  glimpse 


V  J 


1  OR  scientist 

Collaboration  with  the  Royal  Military  College: 

-  Prof.  Greg  Phillips  (Comp.  Eng.) 

•  1  undergrad  student  for  2007-2008 

-  Prof.  Robert  St-John  (Psychology) 

-  Prof.  Allister  MacIntyre  (Psychology) 

•  1  undergrad  student  for  2007-2008 

Defence  R&D  Canada  -  CORA  •  R  &  D  pour  la  defense  Canada  -  CARO 
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dstl  STORM 

L  J  Socio-cultural  Teamworking  for  OR  Models 

HFM-143:  30-31  May  2007 
Carol  Cooper  Chapman 

POC:  Dr  Beejal  Mistry 

bmistry@dstl.gov.uk 


STORM 

•  Impact  of  social  and  cultural  factors 

•  Coalition  NEC  context 

•  Investigate  issues  of  agility 
•Able  to  integrate  with  C2  models. 

•  Support  investment  decisions 


dstl 


30-31  May  2007 
©  Dstl  2007 


Unclassified 


Dstl  is  part  of  the 
Ministry  of  Defence 
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Adjourning 


Transforming 


//  _ _ 

/ 

Goal 

Role,  Task, 
Schedule 


■— «  H;i  ;  ;  ;  i  j  j  -  -Iv . . 

- - '«-■! - - - s  *  -  " •v^'- - ~c - - - : 


Team  Business 
Rules 


V  i 
1 — !S==-:+ — 


V 


Relationships, 

Dependencies 


% 


Linking  knowledge  enablers 
with  team  maturity 


T'v'y»,  !  Decision 
Drivers 


dstl 


30-31  May  2007 
©  Dstl  2007 


Unclassified 


cb 

^_£_\  Dstl  is  part  of  the 
Ministry  of  Defence 


Requisite  modelling 


•  Even  if  modelling  human  behaviour  and  its 
antecedents  is  difficult,  this  is  not  an  adequate 
reason  not  to. 


•  Even  though  we  may  know  little  about  the  inter¬ 
relationships  between  key  variables.  This  is  not 
an  adeauate  reason  to  omit  them. 
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Uncertainty  in  relationships 


Uncertainty  about  relationships  between 
variables  handled  through: 

-  Generic  parametric  functions 

-  Default:  half-period  sine  wave. 

-  Substitute  other  functions  based  on  evidence  from 
sensitivity  analysis  or  empirical  validation. 


dstl 


Unclassified 


STORM  architecture 


Personal/team 


Dynamic 

Teamworking 


Environmental 

changes 


Adjusted 

MOEs 


STORM 

algorithm 


/  coherency 

r-  ^Task  Specific 

Organisational 


Dws 


Team  Efficiency 


Input  data 


dstl 


Unclassified 


RTO-MP-HFM-1 43 


A3  -95 


APPENDIX  3  -  PRESENTATION  SLIDES 


ORGANIZATION 


Consider  2  Teams 


HQ- 

Composition 

Prior 

Exercise 

HQ- 

Location 

Commander - 

Leadership  Style 

Situation 

Information 

Well 

Performing 

Team 

UK/Aus 

Considerable 

exercising 

together 

Co-located 

Well  suited  to 
coalition  ops 

Good 

Poorly 

Performing 

Team 

UK/Aus/ 

Host  Nation 

No  exercising 
together 

Distributed 

Not  suited  to 
coalition  ops 

Patchy 

Nationality  j 

Ways  of  > 

working  / 

AAA 

-eamwork  TVPes  of\  LeadershipX 

Skills  \  interaction  \  Style  \ 

1 

Amount/Quality 
of  Information 

\  \  \ 

Organisational  Direct  Task  Social  Communication 

structures  experience  Skills  Interaction  Style 

dstl 

30-31  May  2007 

©  Dstl  2007 

Unclassified  A 

\  Dstl  is  part  of  the 

pQ  Ministry  of  Defence 

Results 


Well  Performing  Team 

-  Coalition  HQ  well  integrated 

-  Collaboration  timely  -  able  to  take  out  the  threat. 

Poor  Performing  Team 

-  Coalition  HQ  not  well  integrated  well 

-  Collaboration  delayed  -  missed  opportunity  to  take  out  the 
threat,  resulting  in  attacks  on  refugee  camps. 


dstl 
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Achievement 

Move  towards  dynamic  representation  of  the  impact 
of  social  and  cultural  factors  affecting  performance. 


Dr  Beejal  Mistry 
bmistry@dstl.gov.uk 


dstl 


30-31  May  2007 
©  Dstl  2007 


Unclassified 


Dstl  is  part  of  the 
Ministry  of  Defence 
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